Color filter, image processing apparatus, image processing method, image-capture apparatus, image-capture method, program and recording medium

ABSTRACT

A color filter allows a light signal to pass through by each pixel and be incident on an imaging device. The light signal is inputted through a lens and including one of plural different spectral components. The plural different spectral components include a first spectral component which has a widest frequency bandwidth among the plural different spectral components, a second spectral component corresponding to a predetermined frequency band close to a frequency that causes no chromatic aberration of the lens, and a third spectral component expressed in terms of a linear sum of a value resulting from multiplying the first spectral component by a first weighting factor and a value resulting from multiplying the second spectral component by a second weighting factor.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a color filter, an image processingapparatus, an image processing method, an image-capture apparatus, animage-capture method, a program and a recording medium, and moreparticularly, to a color filter, an image processing apparatus, an imageprocessing method, an image-capture apparatus, an image-capture method,a program and a recording medium, which allow to obtain chromaticaberration-cancelled and blur-free images.

2. Description of Related Art

In recent years, digital cameras are widely and commonly used. In thedigital cameras, after an image signal, which is obtained using animaging device (or an image sensor) such as CCDs and CMOSs through alens, is digitized and then given appropriate image processing, aresultant image signal is recorded in a recording medium such as a flashmemory or is directly transferred to an information processing apparatussuch as a personal computer by means of cable connections, infraredcommunications, or the like. With respect to the information processingapparatus such as the personal computers, it is possible to display animage corresponding to the supplied image signal on a monitor such asCRTs (Cathode Ray Tubes) and liquid crystal displays.

An image sensor of a typical type widely used in related art is equippedwith three types of color filters, R (red), G (green) and B (blue).Specifically, each pixel of the image sensor is configured for receivingonly one of R (red), G (green) and B (blue) wavelength components. Inother words, the image signal obtained using the image sensor includes apixel group capable of acquiring a R (red) spectral component, a pixelgroup capable of acquiring a G (green) spectral component and a pixelgroup capable of acquiring a B (blue) spectral component. An example ofcolor arrangement used in the color filter is shown in FIG. 1, beingreferred to as Bayer array.

If such an image-capture processing using the image sensor is to performand a subject to be image-captured is in low brightness, i.e. the lightamount obtained from the subject is insufficient, an output from theimage sensor becomes small, and a resultant image signal is buried undernoise, whereby the final output image also becomes an image containingmuch noise.

Accordingly, in the related art, a signal processing technology is used,which provides an image of high resolution using an imaging devicehaving the color array shown in FIG. 2, or the Bayer array combining Y(white) serving as a luminance signal and R (red), G (green) and B(blue) (See Japanese Patent Application Publication Hei 04-88784, forinstance). According to the technology of this type, assuming that Ypixels are arrayed in mosaic form as shown in FIG. 2 in a color filterarray and that all the pixels are insensitive to infrared light, it ispossible to achieve high resolution using the Y pixels included inpixels arrayed in mosaic form.

Specifically, according to the color filter array shown in FIG. 2, the Ypixels arrayed in mosaic form are sensitive to the substantially wholevisible light, so that it is possible to obtain an image signal ofhigher intensity with respect to the same subject, as compared with thecolor filter array whose G (green) pixels are arrayed in mosaic form asshown in FIG. 1. For that reason, the use of the color filter shown inFIG. 2 may increase S/N ratio of a signal that corresponds to the mosaicarrangement and controls the resolution, as compared with the use of thecolor filter whose G (green) pixels are arrayed in mosaic form as shownin FIG. 1.

SUMMARY OF THE INVENTION

A lens focuses incident light on the image sensor by taking advantage ofa light refraction effect. Because of the use of the refraction effect,blur called chromatic aberration of the lens is produced depending onthe wavelength of incident light. Typically, focusing is made with alens position set up to ensure that a G (green) component is focused.Thus, an image taken in a focused state is an in-focus image in the G(green) component whereas a blur images in the components other than G(green).

The chromatic aberration is now described with reference to FIG. 3.

Specifically, with respect to an image sensor-mounted image-captureapparatus such as the digital cameras in the related art, a lens 1 andan image sensor 2 having a color filter are placed such that theincident light is focused on a position close to the image sensor 2having the color filter, after being refracted by the lens 1.

Typically, the apparatus is configured such that a spectral component ofG (green) is focused on a position of the image sensor 2. In this case,a spectral component of B (blue), which is shorter in wavelength than G(green), is focused on a position shifted from the image sensor 2 to theside of lens as shown in FIG. 3A. Furthermore, the spectral component ofG (green) is focused on the image sensor 2 as shown in FIG. 3B.Furthermore, a spectral component of R (red), which is longer inwavelength than G (green), or the sum of spectral components of R (red)and IR (infrared) is focused on a position shifted from the image sensor2 to the side opposite to the lens as shown in FIG. 3C. Accordingly, inthe focused state as described the above, the components other than G(green) could not be focused exactly on the image sensor. In otherwords, the B (blue) component shorter in wavelength than G (green) andthe R (red) component longer in wavelength than G (green) and/or the IR(infrared) component form a blur image on the image sensor.

To resolve the issue, it suffices to use a lens whose chromaticaberration is small. However, the lens of this type has a disadvantageof being expensive.

Accordingly, it is desirable to obtain an image while reducing noise andblur effect caused by chromatic aberration even under low luminancecondition without using an expensive lens. The present invention hasbeen undertaken in view of the above circumstances.

A color filter according to a first embodiment of the present inventionis a color filter that allows a light signal to pass through by eachpixel and be incident on an imaging device, the light signal beinginputted through a lens and including one of plural different spectralcomponents. The plural different spectral components includes a firstspectral component having a widest frequency bandwidth among the pluraldifferent spectral components, a second spectral component correspondingto a predetermined frequency band close to a frequency that causes nochromatic aberration of the lens, and a third spectral componentexpressed in terms of a linear sum of a value resulting from multiplyingthe first spectral component by a first weighting factor and a valueresulting from multiplying the second spectral component by a secondweighting factor.

The first spectral component may include at least an infrared component.

The first spectral component may include an infrared component and allfrequency bands of visible light.

The first spectral component may include all frequency bands of visiblelight.

The second spectral component may be a frequency component having apredetermined range that corresponds to a green component.

The third spectral component may be a spectral component resulted byexcluding the second spectral component from the first spectralcomponent.

The pixels corresponding to the second spectral component may bearranged in every other pixel in all of a 0-degree direction, a45-degree direction, a 90-degree direction and a 135-degree directionassuming that one array direction of a plane, on which the pixels arearrayed, is set as the 0-degree direction.

The plural different spectral components are of five types of spectralcomponents including the first spectral component, the second spectralcomponent and the third spectral component, wherein the pixelscorresponding to the first spectral component and the third spectralcomponent and the pixels corresponding to a fourth spectral componentand a fifth spectral component may be respectively arranged adjacent tothe pixel corresponding to the second spectral component in one of a0-degree direction, a 45-degree direction, a 90-degree direction and a135-degree direction assuming that one array direction of a plane, onwhich the pixels are arrayed, is set as the 0-degree direction, and thepixels corresponding to the first spectral component and the thirdspectral component may be arrayed in a direction orthogonal to the pixelcorresponding to the second spectral component, and the pixelscorresponding to the fourth spectral component and the fifth spectralcomponents may be arrayed in a direction orthogonal to the pixelcorresponding to the second spectral component.

In the first embodiment of the present invention, the plural differentspectral components with respect to the color filter may include a firstspectral component having a widest frequency bandwidth among the pluraldifferent spectral components, a second spectral component correspondingto a predetermined frequency band close to a frequency that causes nochromatic aberration of the lens, and a third spectral componentexpressed in terms of a linear sum of a value resulting from multiplyingthe first spectral component by a first weighting factor and a valueresulting from multiplying the second spectral component by a secondweighting factor.

An image processing apparatus according to a second embodiment of thepresent invention is an image processing apparatus for receiving a lightsignal and generating image data based on the light signal for eachpixel, the light signal being acquired by each pixel by inputting to apredetermined color filter through a lens and including one of pluraldifferent spectrum component. The apparatus includes contrast componentoperating means for operating a contrast component of the image databased on a first pixel value corresponding to the light signal on apredetermined pixel and a second pixel value resulting frominterpolation processing performed using a pixel value of a pixeladjacent to the predetermined pixel. The contrast component operatingmeans obtains the contrast component of the image data based onoperational processing by the use of, between the first pixel value andthe second pixel value, a pixel value corresponding to a first spectralcomponent having a widest frequency bandwidth among the plural differentspectral components, a pixel value corresponding to a second spectralcomponent corresponding to a predetermined frequency band close to afrequency that causes no chromatic aberration of the lens, or a pixelvalue corresponding to a third spectral component expressed in terms ofa linear sum of a value resulting from multiplying the first spectralcomponent by a first weighting factor and a value resulting frommultiplying the second spectral component by a second weighting factor.

The image data contrast component operated by the contrast componentoperating means may include a pixel value corresponding to the secondspectral component.

The second spectral component may be a frequency component having apredetermined range that corresponds to a green component.

The contrast component operating means may include green componentcalculating means for calculating a pixel value of each pixelcorresponding to a green component based on the first pixel value or thesecond pixel value, red component calculating means for calculating apixel value of each pixel corresponding to a red component based on apixel value corresponding to a predetermined frequency componentcorresponding to the red component among the plural different spectralcomponents and a result of calculation by the green componentcalculating means, and blue component calculating means for calculatinga pixel value of each pixel corresponding to a blue component based on apixel value corresponding to a predetermined frequency componentcorresponding to the blue component among the plural different spectralcomponents and the result of calculation by the green componentcalculating means.

The image processing apparatus may further include pattern directionestimating means for estimating a pattern direction in a vicinity ofeach pixel of the image data, wherein the contrast component operatingmeans may operate the contrast component of the image data based on anestimation result in the pattern direction in the vicinity of each pixelby the pattern direction estimating means.

The image processing apparatus may further include pattern probabilitycalculating means for calculating probability of having a pattern ineach of a 0-degree direction, a 45-degree direction, a 90-degreedirection and a 135-degree direction assuming that one array directionof a plane, on which the pixels are arrayed, is set as the 0-degreedirection, wherein the pattern direction estimating means may estimatethe pattern direction in the vicinity of each pixel of the image databased on a result of calculation by the pattern probability calculatingmeans.

The pattern direction estimating means may decide, based on the resultof calculation by the pattern probability calculating means, whichdirection has higher possibility of being close to the pattern directionin the vicinity of each pixel of the image data between the 0-degreedirection or the 90-degree direction or between the 45-degree directionor the 135-degree direction.

The first spectral component may include at least an infrared component.

The first spectral component may include an infrared component and allfrequency bands of visible light.

The first spectral component may include all frequency bands of visiblelight.

The third spectral component may be a spectral component resulted byexcluding the second spectral component from the first spectralcomponent.

The pixels corresponding to the second spectral component may bearranged in every other pixel in all of a 0-degree direction, a45-degree direction, a 90-degree direction and a 135-degree directionassuming that one array direction of a plane, on which the pixels arearrayed, is set as the 0-degree direction.

The plural different spectral components are of five types of spectralcomponents including the first spectral component, the second spectralcomponent and the third spectral component, wherein the pixelscorresponding to the first spectral component and the third spectralcomponent and the pixels corresponding to a fourth spectral componentand a fifth spectral component may be respectively arranged adjacent tothe pixel corresponding to the second spectral component in one of a0-degree direction, a 45-degree direction, a 90-degree direction and a135-degree direction assuming that one array direction of a plane, onwhich the pixels are arrayed, is set as the 0-degree direction, and thepixels corresponding to the first spectral component and the thirdspectral component may be arrayed in a direction orthogonal to the pixelcorresponding to the second spectral component, and the pixelscorresponding to the fourth spectral component and the fifth spectralcomponent may be arrayed in a direction orthogonal to the pixelcorresponding to the second spectral component.

An image processing method according to the second embodiment of thepresent invention is an image processing method for an image processingapparatus for receiving a light signal and generating image data basedon the light signal for each pixel, the light signal being acquired byeach pixel by inputting to a predetermined color filter through a lensand including one of plural different spectrum component. The methodincludes: acquiring a first pixel value corresponding to the lightsignal on a predetermined pixel; acquiring a second pixel value byperforming interpolation processing using a pixel value of a pixeladjacent to the predetermined pixel; and obtaining a contrast componentof the image data based on operational processing by using, between thefirst pixel value and the second pixel value, a pixel valuecorresponding to a first spectral component having a widest frequencybandwidth among the plural different spectral components, a pixel valuecorresponding to a second spectral component corresponding to apredetermined frequency band close to a frequency that causes nochromatic aberration of the lens, or a pixel value corresponding to athird spectral component expressed in terms of a linear sum of a valueresulting from multiplying the first spectral component by a firstweighting factor and a value resulting from multiplying the secondspectral component by a second weighting factor.

A program according to the second embodiment of the present invention isa program is a program executable by a computer for controllingprocessing of capturing an image and causing the computer to perform theprocessing including the steps of: controlling an operation to acquire alight signal, which is to be obtained, upon receipt of light inputtedvia a lens, as plural different spectral components for each pixelthrough a predetermined color filter; controlling an operation to covertthe acquired light signal into a digital signal; controlling anoperation to acquire a first pixel value corresponding to the lightsignal on a predetermined pixel from the converted digital signal;calculating a second pixel value by performing interpolation processingusing a pixel value of a pixel adjacent to the predetermined pixel;obtaining a contrast component of the image data based on operationalprocessing by using, between the first pixel value and the second pixelvalue, a pixel value corresponding to a first spectral component havinga widest frequency bandwidth among the plural different spectralcomponents, a pixel value corresponding to a predetermined frequencyband close to a frequency that causes no chromatic aberration of thelens, or a pixel value corresponding to a third spectral componentexpressed in terms of a linear sum of a value resulting from multiplyingthe first spectral component by a first weighting factor and a valueresulting from multiplying the second spectral component by a secondweighting factor.

In the second embodiment of the present invention, a first pixel valuecorresponding to a light signal on a predetermined pixel is acquired, asecond pixel value is acquired by performing interpolation processingusing a pixel value of a pixel adjacent to the predetermined pixel, anda contrast component of image data is obtained based on operationalprocessing by the use of, between the first pixel value and the secondpixel value, a pixel value corresponding to a first spectral componenthaving a widest frequency bandwidth among plural different spectralcomponents, a pixel value corresponding to a second spectral componentcorresponding to a predetermined frequency band close to a frequencythat causes no chromatic aberration of the lens, or a pixel valuecorresponding to a third spectral component expressed in terms of alinear sum of a value resulting from multiplying the first spectralcomponent by a first weighting factor and a value resulting frommultiplying the second spectral component by a second weighting factor.

An image-capture apparatus according to a third embodiment of thepresent invention relates to an image-capture apparatus for capturing animage. The apparatus includes: light signal acquiring means foracquiring, for each pixel through a predetermined color filter, lightinputted via a lens as an light signal having plural different spectralcomponents; converting means for converting the light signal acquired bythe light signal acquiring means into a digital signal; and imageprocessing means for processing the digital signal converted by theconverting means to generate image data in which a set of pixel valuesrespectively corresponding to predetermined plural color components isdetermined for all pixels with respect to all the pixels. The imageprocessing means includes contrast component operating means foroperating a contrast component of the image data based on a first pixelvalue corresponding to the light signal on a predetermined pixel, and asecond pixel value resulting from interpolation processing performedusing a pixel value of a pixel adjacent to the predetermined pixel. Thecontrast component operating means obtains the contrast component of theimage data based on operational processing by the use of, between thefirst pixel value and the second pixel value, a pixel valuecorresponding to a first spectral component having a widest frequencybandwidth among the plural different spectral components, a pixel valuecorresponding to a second spectral component corresponding to apredetermined frequency band close to a frequency that causes nochromatic aberration of the lens, or a pixel value corresponding to athird spectral component expressed in terms of a linear sum of a valueresulting from multiplying the first spectral component by a firstweighting factor and a value resulting from multiplying the secondspectral component by a second weighting factor.

The image data contrast component operated by the contrast componentoperating means may include a pixel value corresponding to the secondspectral component.

The second spectral component may be a frequency component having apredetermined range that corresponds to a green component.

The contrast component operating means may include green componentcalculating means for calculating a pixel value of each pixelcorresponding to a green component based on the first pixel value or thesecond pixel value, red component calculating means for calculating apixel value of each pixel corresponding to a red component based on apixel value corresponding to a predetermined frequency componentcorresponding to the red component among the plural different spectralcomponents and a result of calculation by the green componentcalculating means, and blue component calculating means for calculatinga pixel value of each pixel corresponding to a blue component based on apixel value corresponding to a predetermined frequency corresponding tothe blue component among the plural different spectral components andthe result of calculation by the green component calculating means.

The image processing means may further include pattern directionestimating means for estimating a pattern direction in the vicinity ofeach pixel of the image data, wherein the contrast component operatingmeans may operate the contrast component of the image data based on anestimation result of the pattern direction in the vicinity of each pixelby the pattern direction estimating means.

The image processing means may further include pattern probabilitycalculating means for calculating probability of having a pattern ineach of a 0-degree direction, a 45-degree direction, a 90-degreedirection and a 135-degree direction assuming that one array directionof a plane, on which the pixels are arrayed, is set as the 0-degreedirection, wherein the pattern direction estimating means may estimatethe pattern direction in the vicinity of each pixel of the image databased on a result of calculation by the pattern probability calculatingmeans.

The pattern direction estimating means may decide, based on the resultof calculation by the pattern probability calculating means, whichdirection has higher possibility of being close to the pattern directionin the vicinity of each pixel of the image data between the 0-degreedirection or the 90-degree direction or between the 45-degree directionor the 135-degree direction.

The first spectral component may include at least an infrared component.

The first spectral component may include an infrared component and allfrequency bands of visible light.

The first spectral component may include all frequency bands of visiblelight.

The third spectral component may be a spectral component resulted byexcluding the second spectral component from the first spectralcomponent.

The pixels corresponding to the second spectral component may bearranged in every other pixel in all of a 0-degree direction, a45-degree direction, a 90-degree direction and a 135-degree directionassuming that one array direction of a plane, on which the pixels arearrayed, is set as the 0-degree direction.

The plural different spectral components are of five types of spectralcomponents including the first spectral component, the second spectralcomponent and the third spectral component, wherein the pixelscorresponding to the first spectral component and the third spectralcomponent and the pixels corresponding to a fourth spectral componentand a fifth spectral component may be respectively arranged adjacent tothe pixel corresponding to the second spectral component in one of a0-degree direction, a 45-degree direction, a 90-degree direction and a135-degree direction assuming that one array direction of a plane, onwhich the pixels are arrayed, is set as the 0-degree direction, and thepixels corresponding to the first spectral component and the thirdspectral component may be arrayed in a direction orthogonal to the pixelcorresponding to the second spectral component, and the pixelscorresponding to the fourth spectral component and the fifth spectralcomponent may be arrayed in a direction orthogonal to the pixelcorresponding to the second spectral component.

An image-capture method according to the third embodiment of the presentinvention is an image-capture method for an image-capture apparatus forcapturing an image. The method includes: acquiring, for each pixelthrough a predetermined color filter, light inputted via a lens as alight signal having plural different spectral components; converting theacquired light signal into a digital signal; acquiring a first pixelvalue corresponding to the light signal on a predetermined pixel fromthe converted digital signal; acquiring a second pixel value byperforming interpolation processing using a pixel value of a pixeladjacent to the predetermined pixel; obtaining a contrast component ofthe image data based on operational processing by using, between thefirst pixel value and the second pixel value, a pixel valuecorresponding to a first spectral component having a widest frequencybandwidth among the plural different spectral components, a pixel valuecorresponding to a second spectral component corresponding to apredetermined frequency band close to a frequency that causes nochromatic aberration of the lens, or a pixel value corresponding to athird spectral component expressed in terms of a linear sum of a valueresulting from multiplying the first spectral component by a firstweighting factor and a value resulting from multiplying the secondspectral component by a second weighting factor.

A program according to the third embodiment of the present invention isa program executable by a computer for controlling processing ofcapturing an image and causing the computer to perform the processingincluding the steps of: controlling an operation to acquire a lightsignal, which is to be obtained, upon receipt of light inputted via alens, as plural different spectral components for each pixel through apredetermined color filter; controlling an operation to covert theacquired light signal into a digital signal; controlling an operation toacquire a first pixel value corresponding to the light signal on apredetermined pixel from the converted digital signal; calculating asecond pixel value by performing interpolation processing using a pixelvalue of a pixel adjacent to the predetermined pixel; obtaining acontrast component of the image data based on operational processing byusing, between the first pixel value and the second pixel value, a pixelvalue corresponding to a first spectral component having a widestfrequency bandwidth among the plural different spectral components, apixel value corresponding to a predetermined frequency band close to afrequency that causes no chromatic aberration of the lens, or a pixelvalue corresponding to a third spectral component expressed in terms ofa linear sum of a value resulting from multiplying the first spectralcomponent by a first weighting factor and a value resulting frommultiplying the second spectral component by a second weighting factor.

In the third embodiment of the present invention, an light signal, uponreceipt of light inputted via a lens, as plural different spectralcomponents for each pixel through a predetermined color filter isacquired. The acquired light signal is converted into a digital signal.A first pixel value corresponding to the light signal on a predeterminedpixel is acquired from the converted digital signal, a second pixelvalue is calculated through interpolation processing performed using apixel value of a pixel adjacent to the predetermined pixel, and acontrast component of image data is obtained based on operationalprocessing by the use of, between the first pixel value and the secondpixel value, a pixel value corresponding to a first spectral componenthaving a widest frequency bandwidth among the plural different spectralcomponents, a pixel value corresponding to a second spectral componentcorresponding to a predetermined frequency band close to a frequencythat causes no chromatic aberration of the lens, or a pixel valuecorresponding to a third spectral component expressed in terms of alinear sum of a value resulting from multiplying the first spectralcomponent by a first weighting factor and a value resulting frommultiplying the second spectral component by a second weighting factor.

The image-capture apparatus may be in form of an independent apparatusor a block responsible for the image-capture processing of theinformation processing apparatus.

As described the above, according to the first embodiment of the presentinvention, it is possible to provide a predetermined mosaic image.Specifically, with respect to image processing adapted to process theimage data acquired using the color filter of the present invention, itis possible to realize chromatic aberration-cancelled demosaicprocessing using the pixel value of the pixel having the widerfrequency-band spectral component, as compared with the related art.

According to the second embodiment of the present invention, it ispossible to obtain the contrast component. Specifically, it is possibleto perform chromatic aberration-cancelled demosaic processing using thepixel value of the pixel having the wider frequency-band spectralcomponent, as compared with the related art.

According to the third embodiment of the present invention, it ispossible to capture the image. Specifically, with respect to demosaicprocessing, it is possible to provide chromatic aberration-cancelledimage data using the pixel value of the pixel having the widerfrequency-band spectral component, as compared with the related art.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a color filter array in related art;

FIG. 2 illustrates a different type of color filter array in relatedart;

FIG. 3 illustrates chromatic aberration;

FIG. 4 is a block diagram showing a configuration of an image-captureapparatus;

FIG. 5 is a graphic representation of spectral components of a colorfilter;

FIG. 6 illustrates a color array of the color filter shown in FIG. 4;

FIG. 7 illustrates spectral components of the color filter shown in FIG.4;

FIG. 8 is a block diagram showing a configuration of a camera signalprocessing unit shown in FIG. 4;

FIG. 9 illustrates a relation between a pattern direction andinterpolation;

FIG. 10 is a block diagram showing a configuration of a demosaicprocessing unit shown in FIG. 8;

FIG. 11 is a flowchart illustrating image-capture processing;

FIG. 12 is a flowchart illustrating demosaic processing;

FIG. 13 is a flowchart following the flowchart in FIG. 12 to illustratethe demosaic processing;

FIG. 14 is a flowchart illustrating 0-degree directional patternprobability calculation processing;

FIG. 15 is a graphic representation of interpolation processing in a0-degree direction;

FIG. 16 is a flowchart illustrating 45-degree directional patternprobability calculation processing;

FIG. 17 is a graphic representation of interpolation processing in a45-degree direction;

FIG. 18 is a flowchart illustrating 90-degree directional patternprobability calculation processing;

FIG. 19 is a graphic representation of interpolation processing in a90-degree direction;

FIG. 20 is a flowchart illustrating 135-degree directional patternprobability calculation processing;

FIG. 21 is a graphic representation of interpolation processing in a135-degree direction;

FIG. 22 is a flowchart illustrating angular interpolationappropriateness calculation processing;

FIG. 23 is a flowchart following the flowchart in FIG. 22 to illustratethe angular interpolation appropriateness calculation processing;

FIG. 24 is a flowchart illustrating 0-degree direction-interpolated Gcomponent image generation processing;

FIG. 25 is a graphic representation of principle component analysis;

FIG. 26 is a flowchart illustrating 45-degree direction-interpolated Gcomponent image generation processing;

FIG. 27 is a flowchart illustrating 90-degree direction-interpolated Gcomponent image generation processing;

FIG. 28 is a graphic representation of principle component analysis;

FIG. 29 is a flowchart illustrating 135-degree direction-interpolated Gcomponent image generation processing;

FIG. 30 is a flowchart illustrating R component pixel value calculationprocessing;

FIG. 31 is a graphic representation of principle component analysis;

FIG. 32 is a flowchart illustrating B component pixel value calculationprocessing;

FIG. 33 is a graphic representation of principle component analysis;

FIG. 34 illustrates a different type of color array of the color filtershown in FIG. 4; and

FIG. 35 is a block diagram showing a configuration of a personalcomputer.

DETAILED DESCRIPTION OF EMBODIMENTS

An embodiment of the present invention is now described, where oneillustration of a correspondence relation between constituting elementsof the present invention and embodiments contained in the presentspecification or the drawings is given as follows. This is to makecertain that the first embodiments supporting the present invention arecontained in the present specification or the drawings. Thus, even ifthere are any other embodiments, which are not set forth in this sectionas those corresponding to the constituting elements of the presentinvention, but contained in the present specification or the drawings,this should not be interpreted as that the other embodiments arerecognized to be not those corresponding to the constituting elements.Conversely, even if embodiments set forth in this section are given asthose corresponding to the constituting elements, this should not beinterpreted, too, as that the first embodiments are recognized to bethose having no correspondence to constituting elements that are notdescribed therein.

A color filter according to a first embodiment of the present inventionis a color filter (a color filter 22 in FIG. 4, for instance) whichallows a light signal to pass through by each pixel and be incident onan imaging device, the light signal being inputted through a lens andincluding one of plural different spectral components. The pluraldifferent spectral components includes a first spectral component (aspectral component corresponding to W, for instance) that has a widestfrequency bandwidth among the plural different spectral components, asecond spectral component (a spectral component corresponding to G, forinstance) corresponding to a predetermined frequency band close to afrequency that causes no chromatic aberration of the lens, and a thirdspectral component (a spectral component corresponding to M, forinstance) expressed in terms of a linear sum of a value resulting frommultiplying the first spectral component by a first weighting factor anda value resulting from multiplying the second spectral component by asecond weighting factor.

The plural different spectral components are of five types of spectralcomponents including the first spectral component, the second spectralcomponent and the third spectral component, wherein the pixelscorresponding to the first spectral component and the third spectralcomponent and the pixels corresponding to a fourth spectral component (aspectral component corresponding to R, for instance) and a fifthspectral component (a spectral component corresponding to B, forinstance) may be respectively arranged adjacent to the pixelcorresponding to the second spectral component in one of a 0-degreedirection, a 45-degree direction, a 90-degree direction and a 135-degreedirection assuming that one array direction of a plane, on which thepixels are arrayed, is set as the 0-degree direction, and the pixelscorresponding to the first spectral component and the third spectralcomponent may be arrayed in a direction orthogonal to the pixelcorresponding to the second spectral component, and the pixelscorresponding to the fourth spectral component and the fifth spectralcomponent may be arrayed in a direction orthogonal to the pixelcorresponding to the second spectral component.

An image processing apparatus according to a second embodiment of thepresent invention is an image processing apparatus (a demosaicprocessing unit 44 in FIG. 10 or a camera signal processing unit 25 inFIG. 8, for instance) for receiving a light signal and generating imagedata based on the light signal for each pixel, the light signal beingacquired by each pixel by inputting to a predetermined color filterthrough a lens and including one of plural different spectrum component.The apparatus includes contrast component operating means (a 0-degreedirection-interpolated G component image calculation processing unit 71,a 45-degree direction-interpolated G component image calculationprocessing unit 72, a 90-degree direction-interpolated G component imagecalculation processing unit 73, a 135-degree direction-interpolated Gcomponent image calculation processing unit 74 and a G component imagecalculation processing unit 75, or a R component image calculationprocessing unit 81 or a B component image calculation processing unit 82in FIG. 10, for instance) for operating a contrast component of theimage data based on a first pixel value corresponding to the lightsignal on a predetermined pixel, and a second pixel value resulting frominterpolation processing performed using a pixel value of a pixeladjacent to the predetermined pixel. The contrast component operatingmeans obtains the contrast component of the image data based onoperational processing by the use of, between the first pixel value andthe second pixel value, a pixel value corresponding to a first spectralcomponent (the spectral component corresponding to W, for instance)having a widest frequency bandwidth among the plural different spectralcomponents, a pixel value corresponding to a second spectral component(the spectral component corresponding to G, for instance) correspondingto a predetermined frequency band close to a frequency that causes nochromatic aberration of the lens, or a pixel value corresponding to athird spectral component (the spectral component corresponding to M, forinstance) expressed in terms of a linear sum of a value resulting frommultiplying the first spectral component by a first weighting factor anda value resulting from multiplying the second spectral component (thespectral component corresponding to G, for instance) by a secondweighting factor.

The contrast component operating means may include green componentcalculating means (the G component image calculation processing unit 75in FIG. 10, for instance) for calculating a pixel value of each pixelcorresponding to a green component based on the first pixel value or thesecond pixel value, red component calculating means (the R componentimage calculation processing unit 81 in FIG. 10, for instance) forcalculating a pixel value of each pixel corresponding to a red componentbased on a pixel value corresponding to a predetermined frequencycomponent corresponding to the red component among the plural differentspectral components and a result of calculation by the green componentcalculating means, and blue component calculating means (the B componentimage calculation processing unit 82 in FIG. 10, for instance) forcalculating a pixel value of each pixel corresponding to a bluecomponent based on a pixel value corresponding to a predeterminedfrequency component corresponding to the blue component among the pluraldifferent spectral components and the result of calculation by the greencomponent calculating means.

The image processing apparatus may further include pattern directionestimating means (a pattern direction determination unit 65 in FIG. 10,for instance) for estimating a pattern direction in the vicinity of eachpixel of the image data, wherein the contrast component operating meansmay operate the contrast component of the image data based on anestimation result in the pattern direction in the vicinity of each pixelby the pattern direction estimating means.

The image processing apparatus may further include pattern probabilitycalculating means (a 0-degree directional pattern analysis processingunit 51, a 45-degree directional pattern analysis processing unit 52, a90-degree directional pattern analysis processing unit 53 and a135-degree directional pattern analysis processing unit 54 in FIG. 10,for instance) for calculating probability of having a pattern in each ofa 0-degree direction, a 45-degree direction, a 90-degree direction and a135-degree direction assuming that one array direction of a plane, onwhich the pixels are arrayed, is set as the 0-degree direction. Thepattern direction estimating means may estimate the pattern direction inthe vicinity of each pixel of the image data based on a result ofcalculation by the pattern probability calculating means.

The plural different spectral components are of five types of spectralcomponents including the first spectral component, the second spectralcomponent and the third spectral component, wherein the pixelscorresponding to the first spectral component and the third spectralcomponent and the pixels corresponding to a fourth spectral component(the spectral component corresponding to R, for instance) and a fifthspectral component (the spectral component corresponding to B, forinstance) may be respectively arranged adjacent to the pixelcorresponding to the second spectral component in one of a 0-degreedirection, a 45-degree direction, a 90-degree direction and a 135-degreedirection assuming that one array direction of a plane, on which thepixels are arrayed, is set as the 0-degree direction, and the pixelscorresponding to the first spectral component and the third spectralcomponent may be arrayed in a direction orthogonal to the pixelcorresponding to the second spectral component, and the pixelscorresponding to the fourth spectral component and the fifth spectralcomponent may be arrayed in a direction orthogonal to the pixelcorresponding to the second spectral component.

An image processing method according to the second embodiment of thepresent invention is an image processing method applied to an imageprocessing apparatus (the demosaic processing unit 44 in FIG. 10 or thecamera signal processing unit 25 in FIG. 8, for instance) for receivinga light signal and generating image data based on the light signal foreach pixel, the light signal being acquired by each pixel by inputtingto a predetermined color filter through a lens and including one ofplural different spectrum component. The method includes: acquiring afirst pixel value corresponding to the light signal on a predeterminedpixel (processing of Step S41 in FIG. 12, for instance); acquiring asecond pixel value by performing interpolation processing using a pixelvalue of a pixel adjacent to the predetermined pixel (processing ofSteps such as those S244, 246 and 249 in FIG. 24, or the same processingas those in FIGS. 26, 27 and 29, for instance); obtaining a contrastcomponent of the image data based on operational processing by the useof, between the first pixel value and the second pixel value, a pixelvalue corresponding to a first spectral component (the spectralcomponent corresponding to W, for instance) having a widest frequencybandwidth among the plural different spectral components, a pixel valuecorresponding to a second spectral component (the spectral componentcorresponding to G, for instance) corresponding to a predeterminedfrequency band close to a frequency that causes no chromatic aberrationof the lens, or a pixel value corresponding to a third spectralcomponent (the spectral component corresponding to M, for instance)expressed in terms of a linear sum of a value resulting from multiplyingthe first spectral component by a first weighting factor and a valueresulting from multiplying the second spectral component (the spectralcomponent corresponding tog, for instance) by a second weighting factor(processing in FIG. 24 or those in FIGS. 26, 27, 29, 30 and 32, forinstance).

A program according to the second embodiment of the present invention isa program executable by a computer for controlling processing ofreceiving a light signal and generating image data based on the lightsignal for each pixel, the light signal being acquired by each pixel byinputting to a predetermined color filter through a lens and includingone of plural different spectrum component. The program causes thecomputer to perform processing including the steps of: controlling anoperation to acquire a first pixel value corresponding to the lightsignal on a predetermined pixel (processing of Step S41 in FIG. 12, forinstance); calculating a second pixel value by performing interpolationprocessing using a pixel value of a pixel adjacent to the predeterminedpixel (processing of Steps such as those S244, S246 and S249 in FIG. 24or the same processing as those in FIGS. 26, 27 and 29, for instance);and obtaining a contrast component of the image data based onoperational processing by the use of, between the first pixel value andthe second pixel value, a pixel value corresponding to a first spectralcomponent (the spectral component corresponding to W, for instance)having a widest frequency bandwidth among the plural different spectralcomponents, a pixel value corresponding to a second spectral component(the spectral component corresponding to G, for instance) correspondingto a predetermined frequency band close to a frequency that causes nochromatic aberration of the lens, or a pixel value corresponding to athird spectral component (the spectral component corresponding to M, forinstance) expressed in terms of a linear sum of a value resulting frommultiplying the first spectral component by a first weighting factor anda value resulting from multiplying the second spectral component (thespectral component corresponding to G, for instance) by a secondweighting factor (processing in FIG. 24 or those in FIGS. 26, 27, 29, 30and 31, for instance).

An image-capture apparatus according to a third embodiment of thepresent invention is an image-capture apparatus (an image-captureapparatus 11 in FIG. 4, for instance) for capturing an image, andincludes light signal acquiring means (a solid-state imaging device 23in FIG. 4, for instance) for acquiring, for each pixel through apredetermined color filter (a color filter 22 in FIG. 4, for instance),light inputted via a lens (a lens 21 in FIG. 4, for instance) as a lightsignal having plural different spectral components, converting means (aA/D converting unit 24 in FIG. 4, for instance) for converting the lightsignal acquired by the light signal acquiring means into a digitalsignal, and image processing means (the demosaic processing unit 44 inFIG. 10 or the camera signal processing unit 25 in FIG. 8, for instance)for processing the digital signal converted by the converting means togenerate image data in which a set of pixel values respectivelycorresponding to predetermined plural color components is determined forall pixels with respect to all the pixels. The image processing meansincludes contrast component operating means (the 0-degreedirection-interpolated G component image calculation processing unit 71,the 45-degree direction-interpolated G component image calculationprocessing unit 72, the 90-degree direction-interpolated G componentimage calculation processing unit 73, the 135-degreedirection-interpolated G component image calculation processing unit 74and the G component image calculation processing unit 75 or the Rcomponent image calculation processing unit 81 or the B component imagecalculation processing unit 82 in FIG. 10, for instance) for operating acontrast component of the image data based on a first pixel valuecorresponding to the light signal on a predetermined pixel, and a secondpixel value resulting from interpolation processing performed using apixel value of a pixel adjacent to the predetermined pixel. The contrastcomponent operating means obtains the contrast component of the imagedata based on operational processing by the use of, between the firstpixel value and the second pixel value, a pixel value corresponding to afirst spectral component (the spectral component corresponding to W, forinstance) having a widest frequency bandwidth among the plural differentspectral components, a pixel value corresponding to a second spectralcomponent (the spectral component corresponding to G, for instance)corresponding to a predetermined frequency band close to a frequencythat causes no chromatic aberration of the lens, or a pixel valuecorresponding to a third spectral component (the spectral componentcorresponding to M, for instance) expressed in terms of a linear sum ofa value resulting from multiplying the first spectral component by afirst weighting factor and a value resulting from multiplying the secondspectral component (the spectral component corresponding to G, forinstance) by a second weighting factor.

The contrast component operating means may include green componentcalculating means (the G component image calculation processing unit 75in FIG. 10, for instance) for calculating a pixel value of each pixelcorresponding to a green component based on the first pixel value or thesecond pixel value, red component calculating means (the R componentimage calculation processing unit 81 in FIG. 10, for instance) forcalculating a pixel value of each pixel corresponding to a red componentbased on a pixel value corresponding to a predetermined frequencycomponent corresponding to the red component among the plural differentspectral components and a result of calculation by the green componentcalculating means, and blue component calculating means (the B componentimage calculation processing unit 82 in FIG. 10, for instance) forcalculating a pixel value of each pixel corresponding to a bluecomponent based on a pixel value corresponding to a predeterminedfrequency component corresponding to the blue component among the pluraldifferent spectral components and the result of calculation by the greencomponent calculating means.

The image processing means may further include pattern directionestimating means (the pattern direction determination unit 65 in FIG.10, for instance) for estimating a pattern direction in the vicinity ofeach pixel of the image data, wherein the contrast component operatingmeans may operate the contrast component of the image data based on anestimation result of the pattern direction in the vicinity of each pixelby the pattern direction estimating means.

The image processing means may further include pattern probabilitycalculating means (the 0-degree directional pattern analysis processingunit 51, the 45-degree directional pattern analysis processing unit 52,the 90-degree directional pattern analysis processing unit 53 and the135-degree directional pattern analysis processing unit 54 in FIG. 10,for instance) for calculating probability of having a pattern in each ofa 0-degree direction, a 45-degree direction, a 90-degree direction and a135-degree direction assuming that one array direction of a plane, onwhich the pixels are arrayed, is set as the 0-degree direction, whereinthe pattern direction estimating means may estimate the patterndirection in the vicinity of each pixel of the image data based on aresult of calculation by the pattern probability calculating means.

The plural different spectral components are of five types of spectralcomponents including the first spectral component, the second spectralcomponent and the third spectral component, wherein the pixelscorresponding to the first spectral component and the third spectralcomponent and the pixels corresponding to a fourth spectral component(the spectral component corresponding to R, for instance) and a fifthspectral component (the spectral component corresponding to B, forinstance) may be respectively arranged adjacent to the pixelcorresponding to the second spectral component in one of a 0-degreedirection, a 45-degree direction, a 90-degree direction and a 135-degreedirection assuming that one array direction of a plane, on which thepixels are arrayed, is set as the 0-degree direction, and the pixelscorresponding to the first spectral component and the third spectralcomponent may be arrayed in a direction orthogonal to the pixelcorresponding to the second spectral component, and the pixelscorresponding to the fourth spectral component and the fifth spectralcomponent may be arrayed in a direction orthogonal to the pixelcorresponding to the second spectral component.

An image-capture method according to the third embodiment of the presentinvention is an image-capture method for an image-capture apparatus (theimage-capture apparatus 11 in FIG. 4, for instance) for capturing animage, and includes: acquiring, for each pixel through a predeterminedcolor filter (the color filter 22 in FIG. 4, for instance), lightinputted via a lens (the lens 21 in FIG. 4, for instance) as a lightsignal having plural different spectral components (processing of StepS1 in FIG. 11, for instance); converting the acquired light signal intoa digital signal (processing of Step S2 in FIG. 11, for instance);acquiring, from the converted digital signal, a first pixel valuecorresponding to the light signal on a predetermined pixel (processingof Step S41 in FIG. 12, for instance); acquiring a second pixel value byperforming interpolation processing using a pixel value of a pixeladjacent to the predetermined pixel (processing of Steps such as thoseS244, S246 and S249 in FIG. 24 or the same processing as those in FIGS.26, 27 and 29, for instance); and obtaining a contrast component of theimage data based on operational processing by the use of, between thefirst pixel value and the second pixel value, a pixel valuecorresponding to a first spectral component (the spectral componentcorresponding to W, for instance) having a widest frequency bandwidthamong the plural different spectral components, a pixel valuecorresponding to a second spectral component (the spectral componentcorresponding to G, for instance) corresponding to a predeterminedfrequency band close to a frequency that causes no chromatic aberrationof the lens, or a pixel value corresponding to a third spectralcomponent (the spectral component corresponding to M, for instance)expressed in terms of a linear sum of a value resulting from multiplyingthe first spectral component by a first weighting factor and a valueresulting from multiplying the second spectral component (the spectralcomponent corresponding tog, for instance) by a second weighting factor(processing in FIG. 24 or those in FIGS. 26, 27, 29, 30 and 32, forinstance).

A program according to the third embodiment of the present invention isa program executable by a computer for controlling processing ofcapturing an image, and causing the computer to perform processingincluding the steps of: controlling an operation to acquire a lightsignal, which is to be obtained, upon receipt of light inputted via alens (the lens 21 in FIG. 4, for instance), as plural different spectralcomponents for each pixel through a predetermined color filter (thecolor filter 22 in FIG. 4, for instance) (processing of Step S1 in FIG.11, for instance), controlling an operation to convert the acquiredlight signal into a digital signal (processing of Step S2 in FIG. 11,for instance), controlling an operation to acquire a first pixel valuecorresponding to the light signal on a predetermined pixel from theconverted digital signal (processing of Step S41 in FIG. 12, forinstance), calculating a second pixel value by performing interpolationprocessing using a pixel value of a pixel adjacent to the predeterminedpixel (processing of Steps such as those S244, 246 and 249 or the sameprocessing as those in FIGS. 26, 27 and 29, for instance), obtaining acontrast component of the image data based on operational processing bythe use of, between the first pixel value and the second pixel value, apixel value corresponding to a first spectral component (the spectralcomponent corresponding to W, for instance) having a widest frequencybandwidth among the plural different spectral components, a pixel valuecorresponding to a second spectral component (the spectral componentcorresponding to G, for instance) corresponding to a predeterminedfrequency band close to a frequency that causes no chromatic aberrationof the lens, or a pixel value corresponding to a third spectralcomponent (the spectral component corresponding to M, for instance)expressed in terms of a linear sum of a value resulting from multiplyingthe first spectral component by a first weighting factor and a valueresulting from multiplying the second spectral component (the spectralcomponent corresponding to G, for instance) by a second weighting factor(processing in FIG. 24 or those in FIG. 26, 27, 29, 30 or 32, forinstance).

An embodiment of the present invention is now described with referenceto the drawings.

FIG. 4 is a block diagram showing a configuration of the image-captureapparatus 11 having a solid-state imaging device.

Incident light through the optical lens 21 falls on the color filter 22.

The color filter 22 has an array form, which is described later withreference to FIG. 6, and allows light of a predetermined frequency topass through for each pixel and to be incident on the solid-stateimaging device 23.

The solid-state imaging device 23 detects, among the incident lightthrough the optical lens 11, the light passed through the color filter22 to yield optical energy, and converts the resultant optical energyinto an electric signal by means of photoelectric conversion for eachpixel. An image signal outputted as the electric signal resulting fromthe photoelectric conversion using the solid-state imaging device 23 issupplied to the camera signal processing unit 25, after being convertedinto a digital signal by the A/D converting unit 24.

The camera signal processing unit 25 performs processing such asclipping, gamma correction, white balance correction and demosaicprocessing to the supplied digital signal, and supplies the processedsignal to a display unit 26 or an image compressing unit 27. Thedemosaic processing to be performed by the camera signal processing unit25 is described later in detail.

The display unit 26 includes, for instance, a liquid crystal displayelement, a driver for driving the element and the like, and displays, atneed, an image corresponding to image data resulting from predeterminedprocessing by the camera signal processing unit 25.

The image compressing unit 27 performs compressing processing to thesupplied image signal to reduce data volume of the image signal,converts it into image data of predetermined recording image format, andoutputs the converted image data to a recording unit 28 or an externaloutput unit 29. The recording unit 28 records the converted image data,for instance, in a storage element such as hard disks and semiconductormemories or in a detachable recording medium mounted in a predetermineddriver. The external output unit 29 outputs the converted image data toa different apparatus via wire or radio communications. It should benoted that the image compressing may not be always performed by theimage compressing unit 27 to the signal to be recorded or externallyoutputted. In recent years, the number of pixels of the imaging deviceis increasing, and a demand exists for miniaturization of the apparatusitself. Accordingly it is preferable that the image compressing isapplied to cases where the image data is to be recorded or to beexternally outputted through a predetermined medium.

Spectral characteristics of the color filter 22 shown in FIG. 4 aredescribed with reference to FIG. 5, together with those of a typicaltype of color filter.

With respect to the typical type of color filter having the Bayer arrayas previously described with reference to FIG. 1, for instance, filterscorresponding to B channels are of filters providing high transmittanceof light signals whose wavelength is in the range of about 200 to 300nm, which is equivalent to the wavelength of a blue component. Also,filters corresponding to G channels are of filters providing hightransmittance of light signals whose wavelength is in the range of about450 to 550 nm, which is equivalent to the wavelength of a greencomponent. Further, filters corresponding to R channels are of filtersproviding high transmittance of light signals whose wavelength is in therange of about 550 to 650 nm, which is equivalent to the wavelength of ared component. These RGB corresponding filters have the property ofpassing almost no infrared component having the wavelength of about 700nm or more.

On the other hand, the color filter 22 has five filters capable ofacquiring a light signal containing, in addition to R, G and B spectralcomponents, two more wavelength-band spectral components.

In order to meet cases where a subject is not well lighted, the colorfilter 22 includes, for instance, a filter for acquiring the wholevisible light of R, G and B components or a light signal containingvisible light and an invisible light component such as infrared light,as a filter for obtaining a pixel group (hereinafter referred to as afirst pixel group) capable of acquiring a light signal containing aspectral component whose wavelength band is wider (hereinafter referredto as a first spectral component).

Under low illumination condition, a light source of a type that has alow color temperature and radiates much infrared light is frequentlyused. In addition, the use of invisible light such as the infrared lightas an auxiliary light results in less damage to the atmosphere. From theabove, a need exists for a technology that increases effectiveimage-capture sensitivity under the presence of the light sourcecontaining much invisible light such as the infrared light.

Accordingly, it is preferable to acquire the light signal containing, asthe first spectral component, the visible light and the infrared light.Alternatively, the first spectral component may be the whole visiblelight of the R, G and B components, except the infrared light, orotherwise, a spectral component resulted by excluding only apredetermined wavelength band from a wavelength band containing thevisible light and the infrared light, or a spectral component resultedby excluding only a predetermined wavelength band from the visiblelight. It becomes possible to increase the image-capture sensitivity asmuch as an amount of increase in the wavelength bandwidth of the firstspectral component.

Specifically, even if the color filter 22 uses a filter capable ofacquiring the whole visible light, as the filter for acquiring the lightsignal containing a spectral component having the widest wavelengthbandwidth, it becomes also possible to provide a better SN ratio, ascompared with the color filter in the related art. In this case,satisfactory sensitivity may be obtained as long as there is sufficientillumination. However, no satisfactory sensitivity is attained and ahigh quality image with less noise may not be obtained under theenvironment like a dark place where there is not sufficientillumination, and/or low illumination condition where much infraredlight is contained in the light source, or image-capture condition wherean infrared auxiliary light is used at low illumination. On the otherhand, using a color filter of a type that uses filters for acquiring thelight signal containing the visible light and the invisible lightcomponent such as the infrared light as the filters capable of acquiringthe light signal containing the first spectral component specified asthe spectral component having the widest wavelength bandwidth preferablymakes it possible to generate the high quality image with less noise,even under the environment like the dark place where there is notsufficient illumination, and/or the low illumination condition wheremuch infrared light is contained in the light source or theimage-capture condition where the infrared auxiliary light is used atlow illumination.

The filters for acquiring the light signal containing the visible lightand the invisible light component such as the infrared light have theproperty of passing not only all R, G and B component signals but alsoinfrared component having the frequency of 700 nm or more while havingthe peak in the vicinity of about 530 nm as shown by spectralcharacteristics indicated by W in FIG. 5.

Below, the color filter 22 is described as of filters capable ofacquiring the light signal containing, as the first spectral component,the visible light and the invisible light component (hereinafter alsoreferred to as W) such as the infrared light.

In addition, the color filter 22 may also obtain a different pixel group(hereinafter referred to as a second pixel group) capable of acquiring aspectral component (hereinafter referred to as a second spectralcomponent) resulted by excluding a wavelength component in apredetermined range from the first spectral component.

The second spectral component may be a spectral component resulted byexcluding a wavelength component corresponding to any one of the R, Gand B components from the first spectral component. In order to easilyobtain a focused image, it is preferable that a spectral componentresulted by excluding a component having no chromatic aberration fromthe first spectral component is applied to the second spectralcomponent. From the fact that sensibility to light of wavelengths arounda green component wavelength of 555 nm is the most agreeable to thehuman eyes, a lens is typically set up to ensure that no chromaticaberration occurs in the green component. Thus, it is particularlypreferable that a spectral component resulted by excluding a wavelengthcomponent corresponding to G (green) from the first spectral component,that is, a spectral component corresponding to a color component ofmagenta is applied to the second spectral component.

The second spectral component may be, of course, a spectral componentresulted by excluding a wavelength component corresponding to B (blue)from the first spectral component, that is, a yellow spectral component,or otherwise, a spectral component resulted by excluding a wavelengthcomponent corresponding to R (red) from the first spectral component,that is, a spectral component resulted by excluding a wavelengthcomponent corresponding to C (cyan).

In the following, the color filter 22 is described as of filters capableof acquiring the light signal containing, as the second spectralcomponent, the spectral component resulted by excluding the wavelengthcomponent corresponding to G (green) from the wavelength band of thevisible light and the invisible light component such as the infraredlight, that is, the spectral component (hereinafter also referred to asM) corresponding to M (magenta).

Using the color filter 22 as described the enables the solid-stateimaging device 23 to obtain, in addition to the pixel groups adapted toacquire the respective R, G and B spectral components like the colorfilter in the related art, both the pixel group capable of acquiring thefirst spectral component specified as the spectral component longer inwavelength bandwidth than each of the R, G and B components in the colorfilter of the related art, and the pixel group capable of acquiring thesecond spectral component specified as the spectral component resultedby excluding the wavelength component of the predetermined band from thefirst spectral component so as to ensure that sufficient incident lightfalls on the solid-state imaging device 23 even in cases where thesubject is not well-lighted.

With respect to pixel data obtained in this manner, subtracting thepixel data of the second pixel group from the pixel data of the firstpixel group (in other words, calculating the linear sum of therespective pixel values of the first pixel group and the second pixelgroup based on weights W1 and W1, where the weight W1=1, and the weightW2=(−1), makes it possible to obtain pixel data of a wavelengthcomponent of a predetermined band. When there is no chromatic aberrationin the pixel data of the wavelength component of the predetermined band,it is possible to obtain a clear in-focus image. In other words, withthe lens 21 set up so as to get having no chromatic aberration withrespect to the wavelength of the predetermined band resulting fromsubtracting the pixel data of the second pixel group from the pixel dataof the first pixel group, the use of, as a contrast component, the pixeldata of the wavelength component of the predetermined band, or the pixeldata resulted by excluding the chromatic aberration makes it possible tocreate pixel data having no chromatic aberration, leading to creation ofa chromatic aberration-canceled clear image.

One instance of a color filter array with respect to the color filter 22is described with reference to FIG. 6, together with the pixels obtainedin the solid-state imaging device 23. FIG. 6 shows a part of the filterarray of the color filter 22.

In the solid-state imaging device 23, the pixels adapted to provide adifference in spectral components respectively acquirable by the colorfilter 22 are classified into five categories, pixels indicated by W,pixels indicated by M, pixels indicated by G, pixels indicated by R andpixels indicated by B.

The pixels (or the pixels corresponding to the first pixel group adaptedto acquire the first spectral component) indicated by W are pixels usedto acquire the wide wavelength-band spectral component (or the firstspectral component). Specifically, the pixels indicated by W are, forinstance, pixels capable of acquiring all of the B (blue), G (green), R(red) and IR (infrared) spectral components. It should be noted thatwith respect to the color filter 22, no attempt to give processing tofront surfaces of the corresponding pixels of the solid-state imagingdevice 23 typically results in formation of the pixels capable ofacquiring the first spectral component in the solid-state imaging device23.

The pixels (or the pixels corresponding to the second pixel groupadapted to acquire the second spectral component) indicated by M arepixels used to acquire the spectral component (or the second spectralcomponent) resulted by excluding the wavelength component in thepredetermined range (or a wavelength component in a predetermined rangecorresponding to G (green), for instance) from the first spectralcomponent. Specifically, the pixels indicated by M are, for instance,pixels capable of acquiring the B (blue), R (red) and IR (infrared)spectral components. It should be noted that with respect to the colorfilter 22, fitting color filters that allows to pass the magentacomponent on the front surfaces of the corresponding pixels of thesolid-state imaging device 23 typically results in formation of thepixels capable of acquiring the second spectral component in thesolid-state imaging device 23.

The pixels indicated by G are pixels capable of acquiring the G (green)spectral component. It should be noted that with respect to the colorfilter 22, fitting color filters that allows to pass the greencomponent, together with infrared cut filters, on the front surfaces ofthe corresponding pixels of the solid-state imaging device 23 typicallyresults in formation of pixels capable of acquiring the G (green)spectral component in the solid-state imaging device 23.

The pixels indicated by Rare pixels capable of acquiring the R (red)spectral component. It should be noted that with respect to the colorfilter 22, fitting color filters that allows to pass the red component,together with infrared cut filters, on the front surfaces of thecorresponding pixels of the solid-state imaging device 23 typicallyresults in formation of the pixels capable of acquiring the B (blue)spectral component in the solid-state imaging device 23.

The pixels indicated by B are pixels capable of acquiring the B (blue)spectral component. It should be noted that with respect to the colorfilter 22, fitting color filters that allows to pass the blue component,together with infrared cut filters, on the front surfaces of thecorresponding pixels of the solid-state imaging device 23 typicallyresults in formation of the pixels capable of acquiring the B (blue)spectral component in the solid-state imaging device 23.

As shown in FIG. 6, the color filter 22 has a filter array of a sizewhose minimum unit is in the form of a 4×4 matrix, in which assumingthat a certain pixel indicated by W is set as a reference, or shown inthe drawing as the coordinates (0, 0), the pixels indicated by W arearranged at positions represented by the coordinates (0,0), (2,0), (0,2)and (2,2), the pixels indicated by M are arranged at positionsrepresented by the coordinates (1,1), (1,3), (3,1) and (3,3), the pixelsindicated by G are arranged at positions represented by the coordinates(0,1), (0,3), (2,1) and (2,3), the pixels indicated by B are arranged atpositions represented by the coordinates (1,0) and (3,2), and pixelsindicated by R are arranged at positions represented by the coordinates(3,0) and (1,2).

Specifically, referring to the pixels indicated by G assuming that adirection of X-axis is set as a 0-degree direction with respect to thecolor filter 22, the pixels indicated by G are disposed in every otherpixel in all of the 0-degree direction, a 45-degree direction, a90-degree direction and a 135-degree direction. Both the pixels adjacentto the pixel indicated by G in the 0-degree direction are of the pixelsindicated by M. Both the pixels adjacent to the pixel indicated by G inthe 90-degree direction are of the pixels indicated by W. Both thepixels adjacent to the pixel indicated by G in the 45-degree directionare of the pixels indicated by R or B. Both the pixels adjacent to thepixel indicated by G in the 135-degree direction are of the pixelsindicated by either R or B different from the pixels adjacent in the45-degree direction.

The predetermined processing is performed by the camera signalprocessing unit 25 to the image data resulting from image-capturingusing the color filter 22 and the solid-state imaging device 23 asdescribed the above, permitting a final output image to be obtained.

It should be noted that with respect to the color filter 22, thereversed arrangement of the pixels respectively indicated by W and M isalso adaptable to perform the demosaic processing as described later inthe same manner. In addition, the reversed arrangement of the pixelsrespectively indicated by B and R is also adaptable to perform thedemosaic processing as described later in the same manner.

Five wavelength bands with respect to the color filter 22 previouslydescribed with reference to FIG. 6 are now described with reference toFIG. 7.

In FIG. 7, a horizontal axis represents a wavelength, wherein (a)indicates the B (blue) spectral component, (b) indicates the G (green)spectral component, and (c) indicates the R (red) spectral component orthe sum of R (red) and IR (infrared) spectral components.

Then, the wavelength component (or the first spectral component) to beacquired by the first pixel group indicated by W is assumed to be atotal of wavelength components indicated by (a), (b) and (c) in FIG. 7.The wavelength component (or the second spectral component) to beacquired by the second pixel group indicated by M is assumed to be atotal of wavelength components indicated by (a) and (c) in FIG. 7.

Specifically, subtracting the light signal acquired by the second pixelgroup indicated by M from the light signal acquired by the first pixelgroup indicated by W may provide the G (green) spectral component havingno chromatic aberration as shown by (b) in FIG. 7. The use of, as thecontrast component, the G (green) spectral component pixel data with nochromatic aberration may lead to creation of a chromaticaberration-canceled, in-focus, high-contrast, clear image.

As described the above, the data inputted to the camera signalprocessing unit 25 is of the pixel groups corresponding to therespectively different spectral components indicated by W, M, G, B and Routputted from the solid-state imaging device 23. Specifically, thepixels contained in the image data inputted to the camera signalprocessing unit 25 are made classifiable into the pixel groups indicatedby W, M, G, B and R. Assuming that with respect to the pixel coordinates(X, Y) where each of the pixels indicated by W is placed, X and Y areboth even-numbered, it follows that with respect to the pixelcoordinates (X, Y) where each of the pixels indicated by M is placed, Xand Y are supposed to be both odd-numbered. And, with respect to thepixel coordinates (X, Y) where each of the pixels indicated by G isplaced, X and Y are supposed to be respectively even-numbered andodd-numbered. Further, with respect to the pixel coordinates (X, Y)where each of the pixels indicated by B is placed, X and Y are supposedto be respectively odd-numbered and even-numbered, in which case, X−Y−1takes a multiple of 4. Further, with respect to the pixel coordinates(X, Y) where each of the pixels indicated by R is placed, X and Y aresupposed to be respectively odd-numbered and even-numbered, in whichcase, X−Y−3 takes a multiple of 4.

In other words, when assuming that the pixel position of the certainpixel indicated by W is set as a reference point (0, 0), among thepixels contained in the data inputted to the camera signal processingunit 25, the pixels at the coordinates where X and Y are botheven-numbered correspond to W. The pixels at the coordinates where X andY are both odd-numbered correspond to M. The pixels at the coordinateswhere X is even-numbered and Y is odd-numbered correspond to G. Thepixels at the coordinates where X is odd-numbered, Y is even-numberedand X−Y−1 is the multiple of 4 correspond to B. The pixels at thecoordinates where X is odd-numbered, Y is even-numbered and X−Y−3 is themultiple of 4 correspond to R.

The image data outputted from the camera signal processing unit 25appears as data containing three components, R (red), G (green) and B(blue), with respect to all the pixel positions, or demosaic processedimage data.

FIG. 8 is a block diagram showing a more detailed configuration of thecamera signal processing unit 25. The camera signal processing unit 25includes a clipping unit 41, a gamma correcting unit 42, a white balancecorrecting unit 43 and a demosaic processing unit 44.

The clipping unit 41 checks to see whether or not the pixel value ofeach pixel in the supplied image data falls in a defined range of pixelvalues. When the checked pixel value is less than a lower limit in thedefined range, the clipping unit 41 corrects the pixel value so as toreach the lower limit (or clips to a noise level). On the other hand,when the checked pixel value exceeds an upper limit in the definedrange, the clipping unit 42 corrects the pixel value so as to reach theupper limit (or clips to a saturation level).

The gamma correcting unit 42 performs gamma correction to the pixelvalue of each pixel in an inputted mosaic image.

The white balance correcting unit 43 corrects a white balance, or adifference in color taste depending on the light source used duringshooting.

The demosaic processing unit 44 performs, when any one of the R, G and Bcomponents is absent in the respective pixels in the color filter array,interpolation of the absent color component, or the demosaic processing,based on a supplied mosaic image in a predetermined color filter array,to generate image data in which all the pixels hold respectively the R,G and B pixel values.

In the present embodiment, the image processing is performed by an arrayof units disposed in order of the clipping unit 41, the gamma correctingunit 42, the white balance correcting unit 43 and the demosaicprocessing unit 44. Alternatively, it is also allowable to configuredthe array of units, for instance, in order of the clipping processingunit 41, the demosaic processing unit 44, the gamma correcting unit 42and the white balance correcting unit 43 to perform the imageprocessing. It should be noted that, even if the image processing isperformed in the latter order, the demosaic processing unit 44 asdescribed later may perform the similar demosaic processing.

In the demosaic processing unit 44, it is necessary to, after referringeach pixel contained in the image to as a target pixel, obtain R, G andB values with respect to each pixel (represented as the coordinates (H,K), where H and K are both integers). However, only one data among W, M,G, R and B data exists in the pixel (H, K) of the input data. Thus, thedemosaic processing unit 44 calculates the R, G and B values withrespect to each pixel by performing interpolation processing forestimating the pixel value from the pixels in the vicinity of the pixelrepresented as the coordinates (H, K).

The pixel value resulting from the interpolation processing maysometimes differ depending on that the pixel value of the pixel used forthe interpolation is adjacent in which direction, that is, a directionof interpolation. One specific instance of the interpolation processingis described with reference to FIG. 9.

Below it is assumed that the processing is performed for interpolating atarget pixel 101 using the neighboring pixels with respect to an imageof a type that contains white and black gradation patterns in adirection at a predetermined angle θ, for instance, as shown in FIG. 9.Although the interpolation will take place using the adjacent pixels inpractice, a description is herein given with somewhat spaced pixels soas to ensure that influences of an interpolation result depending on thedirection of the pixels used for the interpolation may appear to anextreme.

It is now assumed that the interpolation processing is performed on anactually white or nearly white-colored target pixel 101 using either thesubstantially same-colored pixels 102, 103 specified as the pixels inthe vicinity of the target pixel or pixels 104 and 105 that, althoughbeing also specified as the pixels in the vicinity of the target pixel,exist at an angle of 90° to the pattern angle θ and are thusdark-colored unlike the target pixel. In this processing, using thepixels 104, 105 for the interpolation processing causes the pixel valueof the pixel 101 to appear darker than it really is. On the other hand,using the pixels 102, 103 for the interpolation processing may provide acorrect result.

Specifically, the interpolation processing needs to be performed usingthe neighboring pixels in an appropriate direction based on the pattern(inclusive of a change in color and lightness) in the vicinity of thetarget pixel.

Thus, the demosaic processing unit 44 calculates numerical valuescorresponding to a possibility that the directions of pattern in thevicinity of the target pixel agree respectively with a 0-degreedirection, a 45-degree direction, a 90-degree direction and a 135-degreedirection, in other words, numerical values corresponding to probabilityor possibility of having a pattern in a certain angle direction. Thecalculated numerical values are referred to as 0-degree directionalpattern probability, 45-degree directional pattern probability,90-degree directional pattern probability and 135-degree directionalpattern probability with respect to each pixel position. The patternprobability is described later in detail.

The demosaic processing unit 44 calculates, based on the 0-degreedirectional pattern probability, the 45-degree directional patternprobability, the 90-degree directional pattern probability and the135-degree directional pattern probability, numerical valuesrepresenting appropriateness of the interpolation processing withrespect to each angle or a possibility that the interpolation processingby the use of the neighboring pixel values with the each angle isappropriate.

In this processing, it is assumed that “the pattern probability” withrespect to each angle is used as an index of accuracy in determiningwhether or not the angle concerned is accurate, without taking anglesother than the one concerned into consideration. On the other hand, itis assumed that “the appropriateness” of the interpolation processingwith respect to each angle is used as an index of accuracy indetermining whether or not the angle concerned is accurate, with takingconsideration of angles other than the one concerned.

The demosaic processing unit 44 is also adaptable to provide G (green)component-only-image data under consideration of the pattern directionby, after calculating, with respect to the G (green) component, aninterpolated G (green) component pixel value resulting from theinterpolation by the use of the neighboring pixel values in the 0-degreedirection, an interpolated G (green) component pixel value resultingfrom the interpolation by the use of the neighboring pixel values in the45-degree direction, an interpolated G (green) component pixel valueresulting from the interpolation by the use of the neighboring pixelvalues in the 90-degree direction and an interpolated G (green)component pixel value resulting from the interpolation by the use of theneighboring pixel values in the 135-degree direction, performingweighted addition that performs weighting based on the appropriatenessof the interpolation with respect to each angle.

Then, the demosaic processing unit 44 calculates, based on the G (green)component image data obtained under consideration of the patterndirection, a coefficient of correlation between the G (green) componentand the R (red) component and a coefficient of correlation between the G(green) component and the B (blue) component, and is followed bycalculating each of the R (red) and B (blue) pixel values. By thiscalculation, each of the R (red), G (green) and B (blue) pixel valueswith respect to each pixel position is obtained, so that the demosaicprocessing unit 44 outputs the each pixel value as a resultant image.

FIG. 10 is a block diagram showing a detailed configuration of thedemosaic processing unit 44 of the camera signal processing unit 24previously described with reference to FIG. 8.

The demosaic processing unit 44 includes the 0-degree directionalpattern analysis processing unit 61, the 45-degree directional patternanalysis processing unit 62, the 90-degree directional pattern analysisprocessing unit 63, the 135-degree directional pattern analysisprocessing unit 64, the pattern direction determination unit 65, the0-degree direction-interpolated G component image calculation processingunit 71, the 45-degree direction-interpolated G component imagecalculation processing unit 72, the 90-degree direction-interpolated Gcomponent image calculation processing unit 73, the 135-degreedirection-interpolated G component image calculation processing unit 74,the G component image calculation processing unit 75, the R componentimage calculation processing unit 81 and the B component imagecalculation processing unit 82.

The 0-degree directional pattern analysis processing unit 61, the45-degree directional pattern analysis processing unit 62, the 90-degreedirectional pattern analysis processing unit 63 and the 135-degreedirectional pattern analysis processing unit 64 perform theinterpolation processing in the respective directions depending on thefilter array of the color filter 22 and based on each pixel value of thepixels existing within a predetermined range from the target pixel. Withrespect to the filter array previously described with reference to FIG.6, assuming that the direction of X-axis is set as the 0-degreedirection, the interpolation processing in the 0-degree direction isperformed on one of the adjacent pixels, G and M, using the other. Theinterpolation processing in the 45-degree direction is performed on oneof the adjacent pixels, W and M, using the other. The interpolationprocessing in the 90-degree direction is performed on one of theadjacent pixels, G and W, using the other. The interpolation processingin the 135-degree direction is performed on one of the adjacent pixels,W and M, using the other. The pattern probability with respect to eachangle is calculated by, after obtaining, in addition to the naturallycontained color component pixel value, the other color component pixelvalue through the interpolation processing with respect to more than onepixel and within the predetermined range, performing principle componentanalysis of these two color component pixel values.

Specifically, the 0-degree directional pattern analysis processing unit61 makes reference to a pixel having a G component pixel value withinthe predetermined range from the target pixel to perform the principlecomponent analysis by, after effecting the interpolation processingusing each pixel value of two M component pixels adjacent to thereference pixel in the 0-degree direction with respect to the X-axis,calculating variance of pixel values in more than one pair of the Gcomponent pixel value and an interpolated pixel value of the Mcomponent. In this analysis, if a high contribution rate of a firstprinciple component is found, it may be stated that the patternprobability in the 0-degree direction is high. It should be noted thatthe 0-degree directional pattern analysis processing unit 61 is alsoadaptable to obtain the pattern probability in the 0-degree direction,likewise, if attempting to make reference to a pixel having a Mcomponent pixel value within the predetermined range from the targetpixel to perform the principle component analysis through theinterpolation processing using each pixel value of two G componentpixels adjacent to the reference pixel in the 0-degree direction withrespect to the X-axis.

The 45-degree directional pattern analysis processing unit 62 makesreference to a pixel having a W component pixel value within thepredetermined range from the target pixel to perform the principlecomponent analysis by, after effecting the interpolation processingusing each pixel value of two M component pixels adjacent to thereference pixel in the 45-degree direction to the X-axis, calculatingvariance of pixel values in more than one pair of the W component pixelvalue and an interpolated pixel value of the M component. In thisanalysis, if a high contribution rate of the first principle componentis found, it may be stated that the pattern probability in the 45-degreedirection is high. It should be noted that the 45-degree directionalpattern analysis processing unit 61 is also adaptable to obtain thepattern probability in the 45-degree direction, likewise, if attemptingto make reference to a pixel having a M component pixel value within thepredetermined range from the target pixel to perform the principlecomponent analysis through the interpolation processing using each pixelvalue of two W component pixels adjacent to the reference pixel in the45-degree direction with respect to the X-axis.

The 90-degree directional pattern analysis processing unit 63 makesreference to a pixel having a G component pixel value within thepredetermined range from the target pixel to perform the principlecomponent analysis by, after effecting the interpolation processingusing each pixel value of two W component pixels adjacent to thereference pixel in the 90-degree direction with respect to the X-axis,calculating variance of pixel values in more than one pair of the Gcomponent pixel value and an interpolated pixel value of the Wcomponent. In this analysis, if a high contribution rate of the firstprinciple component is found, it may be stated that the patternprobability in the 90-degree direction is high. It should be noted thatthe 90-degree directional pattern analysis processing unit 63 is alsoadaptable to obtain the pattern probability in the 90-degree direction,likewise, if attempting to make reference to a pixel having a Wcomponent pixel value within the predetermined range from the targetpixel to perform the principle component analysis through theinterpolation processing using each pixel value of two G componentpixels adjacent to the reference pixel in the 90-degree direction withrespect to the X-axis.

The 135-degree directional pattern analysis processing unit 64 makesreference to a pixel having a W component pixel value within thepredetermined range from the target pixel to perform the principlecomponent analysis by, after effecting the interpolation processingusing each pixel value of two M component pixels adjacent to thereference pixel in the 135-degree direction to the X-axis, calculatingvariance of pixel values in more than one pair of the W component pixelvalue and an interpolated pixel value of the M component. In thisanalysis, if a high contribution rate of the first principle componentis found, it may be stated that the pattern probability in the135-degree direction is high. It should be noted that the 135-degreedirectional pattern analysis processing unit 64 is also adaptable toobtain the pattern probability in the 135-degree direction, likewise, ifattempting to make reference to a pixel having a M component pixel valuewithin the predetermined range from the target pixel to perform theprinciple component analysis through the interpolation processing usingeach pixel value of two W component pixels adjacent to the referencepixel in the 135-degree direction with respect to the X-axis.

As described the above, it is assumed that “the pattern probability”with respect to each angle is used as the index of accuracy indetermining whether or not the angle concerned is accurate as the anglein the pattern direction, without taking angles other than the oneconcerned into consideration. One instance of calculations of therespective pattern probabilities is described later in detail withreference to FIGS. 14 to 21.

The pattern direction determination unit 65 calculates theappropriateness of the 0-degree directional interpolation, theappropriateness of the 45-degree directional interpolation, theappropriateness of the 90-degree directional interpolation and theappropriateness of the 135-degree directional interpolation based on thepattern probability already obtained with respect to each angle by eachof the 0-degree directional pattern analysis processing unit 61, the45-degree directional pattern analysis processing unit 62, the 90-degreedirectional pattern analysis processing unit 63 and the 135-degreedirectional pattern analysis processing unit 64.

In this processing, as described the above, it is assumed that “theappropriateness” of the interpolation processing with respect to eachangle is used as the index of accuracy in determining whether or not theangle concerned is accurate, with taking consideration of angles otherthan the one concerned. One instance of calculations of theappropriateness of the interpolation with respect to each angle isdescribed later in detail with reference to flowcharts in FIGS. 22 and23.

The 0-degree direction-interpolated G component image calculationprocessing unit 71 generates the G (green) component-only-image data byperforming the 0-degree directional interpolation processing withrespect to each pixel having no G component. Specifically, the 0-degreedirection-interpolated G component image calculation processing unit 71acquires the W component for all the pixels by taking advantage of thefact that adding the G and M pixel values permits the W pixel value tobe estimated, because the use of the color filter array previouslydescribed with reference to FIG. 6 ensures that the pixel indicated by Wis adjacent to the pixels indicated by R and B in the 0-degreedirection, and the pixels indicated by G and M are adjacent to eachother in the 0-degree direction. One instance of an operation procedureof this processing is described later in detail with reference to FIGS.24 and 25.

The 45-degree direction-interpolated G component image calculationprocessing unit 72 generates the G (green) component-only-image data byperforming the 45-degree directional interpolation processing withrespect to each pixel having no G component. Specifically, the 45-degreedirection-interpolated G component image calculation processing unit 72acquires the G component for all the pixels by taking advantage of thefact that subtracting the M pixel value from the W pixel value permitsthe G pixel value to be estimated, since the use of the color filterarray previously described with reference to FIG. 6 ensures that thepixel indicated by G is adjacent to the pixels indicated by R and B inthe 45-degree direction, and the pixels indicated by W and M areadjacent to each other in the 45-degree direction. One instance of anoperational procedure of this processing is described later in detailwith reference to FIG. 26.

The 90-degree direction-interpolated G component image calculationprocessing unit 73 generates the G (green) component-only-image data byperforming the 90-degree directional interpolation processing withrespect to each pixel having no G (green) component. Specifically, the90-degree direction-interpolated G component image calculationprocessing unit 73 estimates the M component for all the pixels bytaking advantage of the fact that subtracting the G pixel value from theW pixel value permits the M pixel value to be estimated, since the useof the color filter array previously described with reference to FIG. 6ensures that the pixel indicated by M is adjacent to the pixelsindicated by R and B in the 90-degree direction, and the pixelsindicated by G and W are adjacent to each other in the 90-degreedirection. Then, the G component of the target pixel is calculated by,after extracting the pixels having the G component with respect to aninput signal, among the pixels within the predetermined range in thevicinity of the target pixel, performing the principle componentanalysis of the G and M components contained in more than one extractedpixel. One instance of an operational procedure of this processing isdescribed later in detail with reference to FIGS. 27 and 28.

The 135-degree direction-interpolated G component image calculationprocessing unit 74 generates the G (green) component-only-image data byperforming the 135-degree directional interpolation processing withrespect to each pixel having no G (green) component. Specifically, the135-degree direction-interpolated G component image calculationprocessing unit 74 acquires the G component for all the pixels by takingadvantage of the fact that subtracting the M pixel value from the Wpixel value permits the G pixel value to be estimated, since the use ofthe color filter array previously described with reference to FIG. 6ensures that the pixel indicated by G is adjacent to the pixelsindicated by R and B in the 135-degree direction, and the pixelsindicated by W and M are adjacent to each other in the 135-degreedirection. One instance of an operational procedure of this processingis described later in detail with reference to FIG. 29.

The G component image calculation processing unit 75 calculates the Gcomponent image data under consideration of the pattern direction byperforming weighted addition in such a manner as to give weighting tothe G component image data, which is already generated through therespective directional interpolation processing by the 0-degreedirection-interpolated G component image calculation processing unit 71,the 45-degree direction-interpolated G component image calculationprocessing unit 72, the 90-degree direction-interpolated G componentimage calculation processing unit 73 and the 135-degreedirection-interpolated G component image calculation processing unit 74,based on the 0-degree directional interpolation appropriateness, the45-degree directional interpolation appropriateness, the 90-degreedirectional interpolation appropriateness and the 135-degree directionalinterpolation appropriateness, which are respectively already obtainedthrough the processing by the pattern direction determination unit 65.

The R component image calculation processing unit 81 calculates thecorrelation between the R component and the G component by performingthe principle component analysis based on the R component pixel valuecontained in the image data, together with the G component image dataalready obtained by the G component image calculation processing unit75, followed by calculating the R component image data based on thecalculated correlation. This calculation processing is described laterin detail with reference to FIGS. 30 and 31.

The B component image calculation processing unit 82 calculates thecorrelation between the B component and the G component by performingthe principle component analysis based on the B component pixel valuecontained in the image data, together with the G component image dataalready obtained by the G component image calculation processing unit75, followed by calculating the B component image data based on thecalculated correlation. This calculation processing is described laterin detail with reference to FIGS. 32 and 33.

Imaging processing to be performed by the image-capture apparatus 11 isnow described with reference to a flowchart in FIG. 11.

In Step S1, an image sensor, that is, the color filter 22 and thesolid-state imaging device 23 detect the incident light passed throughthe lens 21, then convert the resultant optical energy into an electricsignal by means of photoelectric conversion for each pixel to acquirethe image data, and then supplies the resultant image data to the A/Dconverting unit 24. The color components that the respective pixels ofthe resultant image data hold are determined depending on the filterarray shown in FIG. 6. The color filter 22 has the filter arraypreviously described with reference to FIG. 6.

In Step S2, the A/D converting unit 24 performs A/D conversion, that is,converts the supplied analog image data into image data in the form ofdigital signal.

In Step S3, the camera signal processing unit 25 performs the clippingwith the clipping processing unit 41, the gamma correction with thegamma correcting unit 42, and the white balance correction with thewhite balance correcting unit 43.

In Step S4, the camera signal processing unit 25 performs the demosaicprocessing as described later with reference to flowcharts in FIGS. 12and 13.

In Step S5, the camera signal processing unit 25 judges whether or notthe completely processed image is to be displayed. When the result ofjudgment in the Step S5 is that the image is not displayed, theprocessing goes on to Step S7.

When the result of judgment in the Step S5 is that the image is to bedisplayed, the camera signal processing unit 25 supplies the completelyprocessed image data to the display unit 26 in Step S6. The display unit26 displays an image corresponding to the supplied image data.

When the result of judgment in the Step S5 is that the image is notdisplayed, or after the completion of the processing in the Step S6, thecamera signal processing unit 25 judges whether or not the completelyprocessed image is to be externally outputted in Step S7. When theresult of judgment in the Step S7 is that the image is not externallyoutputted, the processing goes onto Step S9 as described later.

When the result of judgment in the Step S7 is that the image is to beexternally outputted, the camera signal processing unit 25 supplies thecompletely processed image data to the image compressing unit 37 in StepS8. The image compressing unit 37 compresses the supplied image data atneed, and supplies the compressed image data to the external output unit29. The external output unit 29 externally outputs the supplied imagedata.

When the result of judgment in the Step S7 is that the image is notexternally outputted, or after the completion of the processing in theStep S8, the camera signal processing unit 25 judges whether or not thecompletely processed image is to be recorded in Step S9. When the resultof judgment in the Step S9 is that the image is not recorded, theprocessing is brought to an end.

When the result of judgment in the Step S9 is that the image is to berecorded, the camera signal processing unit 25 supplies the completelyprocessed image data to the image compressing unit 37 in Step S10. Theimage compressing unit 37 compresses the supplied image data at need,and supplies the compressed image data to the recording unit 28. Therecording unit 28 records the supplied image data in the storage elementsuch as hard disks and semiconductor memories or in the detachablerecording medium mounted in the predetermined driver, for instance,leading to the end of the processing.

The processing like the enables the image data containing the pixels,which are capable of acquiring the wide frequency-band light signal, tobe imaged, then given various processing including the demosaicprocessing, and finally displayed, externally outputted or recorded atneed. With respect to the demosaic processing as described later withreference to the flowcharts in FIGS. 12 and 13, the contrast componentis extracted with care not to cause the chromatic aberration.

The demosaic processing to be performed in the Step S4 in FIG. 11 is nowdescribed with reference to the flowcharts in FIGS. 12 and 13.

In Step S41, the demosaic processing unit 44 acquires the image datawhose each pixel holds any one of the R, G, B, W and M component pixelvalues based on the filter array previously described with reference toFIG. 6.

In Step S42, the 0-degree directional pattern probability calculationprocessing as described later with reference to the flowchart in FIG. 14is performed.

Specifically, in the Step S42, the 0-degree directional pattern analysisprocessing unit 61 calculates the contribution rate of the firstprinciple component as the pattern probability in the 0-degree directionby performing the principle component analysis about the 0-degreedirection-interpolated data with respect to the target pixel containedin the output image. A low contribution rate of the first principlecomponent is interpreted as a high possibility that there is adisagreement with the pattern in the 0-degree direction, or that theinterpolation processing by the use of the adjacent pixels in the0-degree direction is not appropriate. On the other hand, a highcontribution rate of the first principle component is interpreted as ahigh possibility that although a change of the pattern may exist in the90-degree direction, there is no change of the pattern at least in the0-degree direction, or that the interpolation processing by the use ofthe adjacent pixels in the 0-degree direction is appropriate.

In Step S43, the 45-degree directional pattern probability calculationprocessing as described later with reference to the flowchart in FIG. 16is performed.

Specifically, in the Step S43, the 45-degree directional patternanalysis processing unit 62 calculates the contribution rate of thefirst principle component as the pattern probability in the 45-degreedirection by performing the principle component analysis about the45-degree direction-interpolated data with respect to the target pixelcontained in the output image. A low contribution rate of the firstprinciple component is interpreted as a high possibility that there is adisagreement with the pattern in the 45-degree direction, or that theinterpolation processing by the use of the adjacent pixels in the45-degree direction is not appropriate. On the other hand, a highcontribution rate of the first principle component is interpreted as ahigh possibility that although a change of the pattern may exist in the135-degree direction, there is no change of the pattern at least in the45-degree direction, or that the interpolation processing by the use ofthe adjacent pixels in the 45-degree direction is appropriate.

In Step S44, the 90-degree directional pattern probability calculationprocessing as described later with reference to the flowchart in FIG. 18is performed.

Specifically, in the Step S44, the 90-degree directional patternanalysis processing unit 63 calculates the contribution rate of thefirst principle component as the pattern probability in the 90-degreedirection by performing the principle component analysis about the90-degree direction-interpolated data with respect to the target pixelcontained in the output image. A low contribution rate of the firstprinciple component is interpreted as a high possibility that there is adisagreement with the pattern in the 90-degree direction, or that theinterpolation processing by the use of the adjacent pixels in the90-degree direction is not appropriate. On the other hand, a highcontribution rate of the first principle component is interpreted as ahigh possibility that although a change of the pattern may exist in the0-degree direction, there is no change of the pattern at least in the90-degree direction, or that the interpolation processing by the use ofthe adjacent pixels in the 90-degree direction is appropriate.

In Step S45, the 135-degree directional pattern probability calculationprocessing as described later with reference to the flowchart in FIG. 20is performed.

Specifically, in the Step S45, the 135-degree directional patternanalysis processing unit 64 calculates the contribution rate of thefirst principle component as the pattern probability in the 135-degreedirection by performing the principle component analysis about the135-degree direction-interpolated data with respect to the target pixelcontained in the output image. A low contribution rate of the firstprinciple component is interpreted as a high possibility that there is adisagreement with the pattern in the 135-degree direction, or that theinterpolation processing by the use of the adjacent pixels in the135-degree direction is not appropriate. On the other hand, a highcontribution rate of the first principle component is interpreted as ahigh possibility that although a change of the pattern may exist in the45-degree direction, there is no change of the pattern at least in the135-degree direction, or that the interpolation processing by the use ofthe adjacent pixels in the 135-degree direction is appropriate.

In Step S46, the angular interpolation appropriateness calculationprocessing as described later with reference to the flowcharts in FIGS.22 and 23 is performed.

Specifically, in the Step S46, the pattern direction determination unit65 respectively calculates, with respect to the target pixel, the0-degree directional interpolation appropriateness, the 45-degreedirectional interpolation appropriateness, the 90-degree directionalinterpolation appropriateness and the 135-degree directionalinterpolation appropriateness based on the four directional patternprobabilities already calculated through the processing in the Steps S42to S45.

In Step S47, the 0-degree direction-interpolated G component imagegeneration processing as described later with reference to the flowchartin FIG. 24 is performed.

Specifically, in the Step S47, the 0-degree direction-interpolated Gcomponent image calculation processing unit 71 generates the G (green)component image data resulting from the 0-degree directionalinterpolation processing.

In Step S48, the 45-degree direction-interpolated G component imagegeneration processing as described later with reference to the flowchartin FIG. 26 is performed.

Specifically, in the Step S48, the 45-degree direction-interpolated Gcomponent image calculation processing unit 72 generates the G (green)component image data resulting from the 45-degree directionalinterpolation processing.

In Step S49, the 90-degree direction-interpolated G component imagegeneration processing as described later with reference to the flowchartin FIG. 27 is performed.

Specifically, in the Step S49, the 90-degree direction-interpolated Gcomponent image calculation processing unit 73 generates the G (green)component image data resulting from the 90-degree directionalinterpolation processing.

In Step S50, the 135-degree direction-interpolated G component imagegeneration processing as described later with reference to the flowchartin FIG. 29 is performed.

Specifically, in the Step S50, the 135-degree direction-interpolated Gcomponent image calculation processing unit 74 generates the G (green)component image data resulting from the 135-degree directionalinterpolation processing.

In Step S51, the G component image calculation processing unit 75calculates the G (green) component image data under consideration of thepattern direction by performing weighted addition of the G (green)component image data resulting from the interpolation in the respectivedirections through the processing of the Steps S47 to S50 based on therespective angular correction appropriateness already calculated throughthe processing of the Step S46.

Specifically, assuming that the coordinates of the target pixel positionis represented as (H, K), the G component image calculation processingunit 75 calculates, with respect to the target pixel, a value of“0-degree directional interpolation appropriatenes×0-degreedirection-interpolated G component image pixel value+45-degreedirectional interpolation appropriateness×45-degreedirection-interpolated G component image pixel value+90-degreedirectional interpolation appropriateness×90-degreedirection-interpolated G component image pixel value+135-degreedirectional interpolation appropriateness×135-degreedirection-interpolated G component image pixel value” as a pixel valuewith respect to the coordinates (H, K) of the G (green) component imagedata to be obtained under consideration of the pattern direction. Then,this operation is performed with respect to all the pixels, leading togeneration of the G (green) component image data under consideration ofthe pattern direction.

The G (green) component image data generated through the processing isgiven using the data of the W and M components acceptable to light whosefrequency band is wider than that corresponding to any one of the R, Gand B components in the related art. Specifically, the G (green)component image data generated through the processing may provide asatisfactory image even in the cases where the subject is notwell-lighted, as compared with the image generated with only each of theR, G and B components like the related art. In addition, the data of theW and M components is used to generate the G component having nochromatic aberration, so that it is possible to obtain G component imagedata that permits a suppression of an out-of-focus effect caused by thechromatic aberration, although the light in the wide frequency band isaccepted.

In Step S52, the R component pixel value calculation processing asdescribed later with reference to the flowchart in FIG. 30 is performed.

Specifically, in the Step S52, the R component image calculationprocessing unit 81 calculates the correlation between the R componentand the G component by performing the principle component analysis basedon the R component pixel value contained in the input signal, togetherwith the G component image data already calculated by the G componentimage calculation processing unit 75, followed by calculating Rcomponent image data based on the calculated correlation.

In Step S53, the B component pixel value calculation processing asdescribed later with reference to the flowchart in FIG. 32 is performed.

Specifically, in the Step S53, the B component image calculationprocessing unit 82 calculates the correlation between the B componentand the G component by performing the principle component analysis basedon the B component pixel value contained in the input signal, togetherwith the G component image data already calculated by the G componentimage calculation processing unit 75, followed by calculating Bcomponent image data based on the calculated correlation.

Then, in Step S54, the demosaic processing unit 44 outputs, as outputimage data, the G, R and B pixel values already calculated for eachpixel by the G component image calculation processing unit 75, the Rcomponent image calculation processing unit 81 and the B component imagecalculation processing unit 82, and the processing returns to the StepS4 in FIG. 11, and is followed by the Step S5.

The processing like the brings the demosaic processing into realization,and thus may provide image data whose respective pixels hold completelythe R, G and B components, from the mosaic image containing five colorcomponents, R, G, B, W and W.

In addition, the G component image data having no chromatic aberrationis generated using each pixel value of the W and M components wider infrequency band than each of the R, G and B components, or greater inquantity of light detected, as compared with the R, G and B components,followed by generating the R and B component image data based on thegenerated G component image data, so that it is possible to obtain asatisfactory image even in cases where the subject is not well-lighted,for instance.

Further, an attempt is made to perform the interpolation processingunder consideration of the pattern direction contained in the image, sothat it is possible to obtain a more satisfactory image, as comparedwith the interpolation processing by the use of pixel values of theneighboring pixels in all directions without taking the patterndirection into consideration at all.

The 0-degree directional pattern probability calculation processing tobe performed in the Step S42 in FIG. 12 is now described with referenceto the flowchart in FIG. 14.

In Step S81, the 0-degree directional pattern analysis processing unit61 selects one unprocessed target pixel, wherein it is assumed that atarget pixel position is represented as the coordinates (H, K), forinstance

In Step S82, the 0-degree directional pattern analysis processing unit61 detects the G pixels existing within the predetermined range from thetarget pixel.

With respect to the filter array previously described with reference toFIG. 6, for instance, the pixel indicated by G is placed at more thenone position represented by the coordinates (X, Y), where X iseven-numbered, and Y is odd-numbered. Thus, the 0-degree directionalanalysis processing unit 61 detects the G pixels represented by thecoordinates like the above, wherein the predetermined range indicates arange experientially or experimentally determined based on conditionssuch as the number of samples for the principle component analysis andthe closeness of correlation depending on a distance from the targetpixel. Assuming that the target pixel position is represented as thecoordinates (H, K), for instance, the predetermined range may includethose such as a range having a size of 9×9 pixels represented by thecoordinates (X, Y) where H−4≦X≦H+4 and K+4≦Y≦K+4. Alternatively, a rangehaving a different size or a range covering the predetermined number ofpixels selected from the pixels existing close to each other with thetarget pixel as the center may also be used.

In Step S83, the 0-degree directional pattern analysis processing unit61 performs the interpolation using the M pixels adjacent in the0-degree direction to all the detected G pixels (X, Y).

Specifically, with respect to the detected G pixels represented by thecoordinates (X, Y) in the Step S82, the M pixels represented by thecoordinates (X−1, Y) and (X+1, Y) are arranged adjacent in the 0-degreedirection, as shown in FIG. 15. Thus, calculating an average of pixelvalues of the M pixels represented by the two coordinates may provide aninterpolated pixel value of the 0-degree direction-interpolated M pixelwith respect to the coordinates (X, Y). Specifically, the 0-degreedirectional pattern analysis processing unit 61 may calculate, accordingto an equation (1), a 0-degree direction-interpolated M pixel value withrespect to the coordinates (X, Y), where X is even-numbered and Y isodd-numbered.0-degree direction-interpolated M pixel value with respect tocoordinates (X, Y)={(M pixel value at coordinates (X−1, Y))+(M pixelvalue at coordinates (X+1, Y))}÷2  (1)where X is even-numbered and Y is odd-numbered.

In Step S84, the 0-degree directional pattern analysis processing unit61 performs the principle component analysis by plotting, on atwo-dimensional space of G and M, the G pixel value, together with theinterpolated M pixel value already obtained in the Step S83, withrespect to each of the pixel positions corresponding to the G pixelsalready detected within the predetermined range in the Step S82.

Specifically, within the predetermined range, the G pixel value existsin the input image at each position represented by the coordinates (X,Y), where X is even-numbered, and Y is odd-numbered, while the 0-degreedirection-interpolated M pixel value at the corresponding position isalready obtained in the Step S83. Based on the fact, a pair of G and Mpixel values is supposed to hold within the predetermined range withrespect to more than one coordinates (X, Y), where X is even-numbered,and Y is odd-numbered. Thus, the 0-degree directional pattern analysisprocessing unit 61 may perform the principle component analysis byplotting more than one pair of G and M pixel values on thetwo-dimensional space of G and M.

In Step S85, the 0-degree directional pattern analysis processing unit61 calculates the contribution rate of the first principle componentwith respect to the principle component analysis in the Step S84 toprovide the 0-degree directional pattern probability.

The contribution rate of the first principle component is found by(Variance value of the first principle component)/(Sum of variances ofeach variable), which is equivalent to (Variance value of the firstprinciple component)/(Amount of scattering of the whole samples).Specifically, for portions where there is no change of the pattern inthe 0-degree direction, the plotted pairs obtained on thetwo-dimensional space of G and M after the interpolation properlyperformed by means of the 0-degree directional interpolation ought tofall on a single straight line. In other words, when there is no changeof the pattern in the 0-degree direction, the result of the principlecomponent analysis shows that the contribution rate of components otherthan the first principle component appears as approximately 0. Thus, thecontribution rate of the first principle component is equivalent to the0-degree directional pattern probability with respect to the targetpixel. If the principle component analysis results in a low contributionrate of the first principle component, it is supposed that the change ofthe pattern exists in the 0-degree direction. Thus, it becomes possibleto estimates that the 0-degree directional interpolation is notadaptable to obtain a correctly interpolated image.

In this processing, It should be noted that the G and M pixel values areassumed to be correlated to each other with respect to local ranges suchas those represented by H−4≦X≦H+4 and K+4≦Y≦K+4. In addition, hueusually remains unchanged at positions where there is the change of thepattern. That is, the G and M pixel values are usually supposed to havea proportional relation (where a coefficient of proportion is positive)with each other. Thus, in calculating the first principle component, itis also allowable to make a condition that it is necessary to passthrough the origin. Alternatively, when the direction of the firstprinciple component forms a negative inclination, it is also allowableto make an attempt to reduce a value of the 0-degree directional patternprobability at a predetermined rate.

In Step S86, the 0-degree directional pattern analysis processing unit61 judges whether or not the 0-degree directional pattern probabilityhas been already calculated for all the pixels. When the result ofjudgment in the Step S86 is that the 0-degree directional patternprobability is not calculated yet for all the pixels, the processingreturns to the Step S81, causing reprocessing to be performed from theStep S81. When the result of judgment in the Step S86 is that the0-degree directional pattern probability has been already calculated forall the pixels, the processing returns to the Step S42 in FIG. 12, andis followed by the Step S43.

The processing like the may calculate the possibility that the directionof the pattern in the vicinity of the target pixel agrees with the0-degree direction, in other words, the probability (or possibility) ofhaving the pattern in the 0-degree direction, using the pixel value ofthe pixel having the G component and the pixel value of the pixel havingthe M component among the pixels in the input image data.

The 45-degree directional pattern probability calculation processing tobe performed in the Step S42 in FIG. 12 is now described with referenceto the flowchart in FIG. 16.

In Step S111, the 45-degree directional pattern analysis processing unit62 selects one unprocessed target pixel, wherein it is assumed that thetarget pixel position is represented as coordinates (H, K), forinstance.

In Step S112, the 45-degree directional pattern analysis processing unit62 detects the W pixels existing within the predetermined range from thetarget pixel.

With respect to the filter array previously described with reference toFIG. 6, for instance, the pixel indicated by W is placed at more thanone position represented by the coordinates (X, Y), where X and Y areboth even-numbered. The 45-degree directional analysis processing unit62 detects the W pixels represented by the coordinates like the.

In Step S113, the 45-degree directional pattern analysis processing unit62 performs the interpolation using the M pixels adjacent in the45-degree direction to all the detected W pixels (X, Y).

Specifically, with respect to the detected W pixels represented by thecoordinates (X, Y) in the Step S112, the M pixels represented by thecoordinates (X−1, Y−1) and (X+1, Y+1) are arranged adjacent in the45-degree direction as shown in FIG. 17. Thus, calculating an average ofpixel values of the M pixels represented by the two coordinates mayprovide an interpolated pixel value of the 45-degreedirection-interpolated M pixel with respect to the coordinates (X, Y).Specifically, the 45-degree directional pattern analysis processing unit62 may calculate, according to an equation (2), a 45-degree directioninterpolated M pixel value with respect to the coordinates (X, Y), whereX and Y are both even-numbered.45-degree direction-interpolated M pixel value with respect tocoordinates (X, Y)={(M pixel value at coordinates (X−1, Y−1))+(M pixelvalue at coordinates (X+1, Y+1))}÷2  (2)where X and Y are both even-numbered

In Step S114, the 45-degree directional pattern analysis processing unit62 performs the principle component analysis by plotting, on atwo-dimensional space of W and M, the W pixel value, together with theinterpolated M pixel value already obtained in the Step S113, withrespect to each of the pixel positions corresponding to the W pixelsalready detected within the predetermined range in the Step S112.

Specifically, within the predetermined range, the W pixel value existsin the input image at each position represented by the coordinates (X,Y), where X and Y are both even-numbered, while the 45-degreedirection-interpolated M pixel value at the corresponding position isalready obtained in the Step S113. Based on the fact, a pair of W and Mpixel values is supposed to hold within the predetermined range withrespect to more than one coordinates (X, Y), where X and Y are botheven-numbered. Thus, the 45-degree directional pattern analysisprocessing unit 62 may perform the principle component analysis byplotting more than one pair of W and M pixel values on thetwo-dimensional space of W and M.

In Step S115, the 45-degree directional pattern analysis processing unit62 calculates the contribution rate of the first principle componentwith respect to the principle component analysis in the Step S114 toprovide the 45-degree directional pattern probability.

The contribution rate of the first principle component is found by(Variance value of the first principle component)/(Sum of variances ofeach variable), which is equivalent to (Variance value of the firstprinciple component)/(Amount of scattering of the whole samples).Specifically, for portions where there is no change of the pattern inthe 45-degree direction, the plotted pairs obtained on thetwo-dimensional space of the W and M components after the interpolationproperly performed by means of the 45-degree directional interpolationought to fall on a single straight line. In other words, when there isno change of the pattern in the 45-degree direction, the result of theprinciple component analysis shows that a contribution rate of thecomponents other than the first principle component appears asapproximately 0. Thus, the contribution rate of the first principlecomponent is equivalent to the 45-degree directional pattern probabilitywith respect to the target pixel. If the principle component analysisresults in a low contribution rate of the first principle component, itis supposed that the change of the pattern exists in the 45-degreedirection. Thus, it becomes possible to assume that the 45-degreedirectional interpolation cannot obtain a correctly interpolated image.

In this processing, It should be noted that the W and M pixel values areassumed to be correlated to each other with respect to local regionssuch as those shown by H−4≦X≦H+4 and K+4≦Y≦K+4. In addition, hue usuallyremains unchanged at positions where there is the change of the pattern.That is, the W and M pixel values are usually supposed to have aproportional relation (where a coefficient of proportion is positive)with each other. Thus, in calculating the first principle component, itis also allowable to make a condition that it is necessary to passthrough the origin. Alternatively, when the direction of the firstprinciple component forms a negative inclination, it is also allowableto make an attempt to reduce a value of the 45-degree directionalpattern probability at a predetermined rate.

In Step S116, the 45-degree directional pattern analysis processing unit62 judges whether or not the 45-degree directional pattern probabilityhas been already calculated for all the pixels. When the result ofjudgment in the Step S116 is that the 45-degree directional patternprobability is not calculated yet for all the pixels, the processingreturns to the Step S111, causing reprocessing to be performed from theStep S111. When the result of judgment in the Step S116 is that the45-degree directional pattern probability has been already calculatedfor all the pixels, the processing returns to the Step S43 in FIG. 12,and is followed by the Step S44.

The processing like the may calculate the possibility that the directionof the pattern in the vicinity of the target pixel agrees with the45-degree direction, in other words, the probability (or possibility) ofhaving the pattern in the 45-degree direction, using the pixel value ofthe pixel having the W component and the pixel value of the pixel havingthe M component among the pixels in the input image data.

The 90-degree directional pattern probability calculation processing tobe performed in the Step S44 in FIG. 12 is now described with referenceto the flowchart in FIG. 18.

In Step S141, the 90-degree directional pattern analysis processing unit63 selects one unprocessed target pixel, wherein it is assumed that thetarget pixel position is represented as the coordinates (H, K), forinstance.

In Step S142, the 90-degree directional pattern analysis processing unit63 detects the G pixels existing within the predetermined range from thetarget pixel.

As described the above, with respect to the filter array previouslydescribed with reference to FIG. 6, for instance, the pixel indicated byG is placed at more than one position represented by the coordinates (X,Y), where X is even-numbered and Y is odd-numbered. The 90-degreedirectional analysis processing unit 63 detects the G pixels representedby the coordinates like the.

In Step S143, the 90-degree directional pattern analysis processing unit63 performs the interpolation using the M pixels adjacent in the90-degree direction to all the detected G pixels (X, Y).

Specifically, with respect to the detected G pixels represented by thecoordinates (X, Y) in the Step S142, the W pixels represented by thecoordinates (X, Y+1) and (X, Y−1) are arranged adjacent in the 90-degreedirection as shown in FIG. 19. Thus, calculating an average of pixelvalues of the W pixels represented by the two coordinates may provide aninterpolated pixel value of the 90-degree direction-interpolated W pixelwith respect to the coordinates (X, Y). Specifically, the 90-degreedirectional pattern analysis processing unit 63 may calculate, accordingto an equation (3), a 90-degree direction-interpolated W pixel valuewith respect to the coordinates (X, Y), where the X is even-numbered andY is odd-numbered.90-degree direction-interpolated W pixel value with respect tocoordinates (X, Y)={(W pixel value at coordinates (X, Y+1))+(W pixelvalue at coordinates (X, Y−1))}÷2  (3)where X is even-numbered and Y is odd-numbered

In Step S144, the 90-degree directional pattern analysis processing unit63 performs the principle component analysis by plotting, on atwo-dimensional space of G and W, the G pixel value, together with aninterpolated W pixel value already obtained in the Step S143, withrespect to each of the pixel positions corresponding to the G pixelsalready detected within the predetermined range in the Step S142.

Specifically, within the predetermined range, the G pixel value existsin the input image at each position represented by the coordinates (X,Y), where X is even-numbered, and Y is odd-numbered, while the 90-degreedirection-interpolated W pixel value at the corresponding position isalready obtained in the Step S143. Based on the fact, a pair of G and Wpixel values is supposed to hold within the predetermined range withrespect to more than one coordinates (X, Y) where X is even-numbered,and Y is odd-numbered. Thus, the 90-degree directional pattern analysisprocessing unit 63 may perform the principle component analysis byplotting more than one pair of G and W pixel values on thetwo-dimensional space of G and W.

In Step S145, the 90-degree directional pattern analysis processing unit63 calculates the contribution rate of the first principle componentwith respect to the principle component analysis in the Step S144 toprovide the 90-degree directional pattern probability.

The contribution rate of the first principle component is found by(Variance value of the first principle component)/(Sum of variances ofeach variable), which is equivalent to (Variance value of the firstprinciple component)/(Amount of scattering of the whole samples).Specifically, for portions where there is no change of the pattern inthe 90-degree direction, the plotted pairs obtained on thetwo-dimensional space of G and W after the interpolation properlyperformed by means of the 90-degree directional interpolation ought tofall on a single straight line. In other words, when there is no changeof the pattern in the 90-degree direction, the result of the principlecomponent analysis shows that a contribution rate of the componentsother than the first principle component appears as approximately 0.Thus, the contribution rate of the first principle component isequivalent to the 90-degree directional pattern probability with respectto the target pixel. If the principle component analysis results in alow contribution rate of the first principle component, it is supposedthat the change of the pattern exists in the 90-degree direction. Thus,it becomes possible to estimates that the 90-degree directionalinterpolation is not adaptable to obtain a correctly interpolated image.

In this processing, It should be noted that the G and W pixel values areassumed to be correlated to each other with respect to local regionssuch as those shown by H−4≦X≦H+4 and K+4≦Y≦K+4. In addition, hue usuallyremains unchanged at positions where there is the change of the pattern.That is, the G and W pixel values are usually supposed to have aproportional relation (where a coefficient of proportion is positive)with each other. Thus, in calculating the first principle component, itis also allowable to make a condition that it is necessary to passthrough the origin. Alternatively, when the direction of the firstprinciple component forms a negative inclination, it is also allowableto make an attempt to reduce a value of the 90-degree directionalpattern probability at a predetermined rate.

In Step S146, the 90-degree directional pattern analysis processing unit63 judges whether or not the 90-degree directional pattern probabilityhas been already calculated for all the pixels. When the result ofjudgment in the Step S146 is that the 90-degree directional patternprobability is not calculated yet for all the pixels, the processingreturns to the Step S141, causing reprocessing to be performed from theStep S141. When the result of judgment in the Step S146 is that the90-degree directional pattern probability has been already calculatedfor all the pixels, the processing returns to the Step S44 in FIG. 12,and is followed by the Step S45.

The processing like the may calculate the possibility that the directionof the pattern in the vicinity of the target pixel agrees with the90-degree direction, in other words, the probability (or possibility) ofhaving the pattern in the 90-degree direction, using the pixel value ofthe pixel having the G component and the pixel value of the pixel havingthe W component among the pixels in the input image data.

The 135-degree directional pattern probability calculation processing tobe performed in the Step S45 in FIG. 12 is now described with referenceto the flowchart in FIG. 20.

In Step S171, the 135-degree directional pattern analysis processingunit 64 selects one unprocessed target pixel, wherein it is assumed thatthe target pixel position is represented as the coordinates (H, K), forinstance.

In Step S172, the 135-degree directional pattern analysis processingunit 64 detects the W pixels existing within the predetermined rangefrom the target pixel.

With respect to the filter array preciously described with reference toFIG. 6, for instance, the pixel indicated by W exists at more than oneposition represented by the coordinates (X, Y), where X and Y are botheven-numbered. The 135-degree directional analysis processing unit 64detects the W pixels represented by the coordinates like the.

In Step S173, the 135-degree directional pattern analysis processingunit 64 performs the interpolation using the M pixels adjacent in the135-degree direction to all the detected W pixels (X, Y).

Specifically, with respect to the detected W pixels represented by thecoordinates (X, Y) in the Step S172, the M pixels represented by thecoordinates (X−1, Y+1) and (X+1, Y−1) are arranged adjacent in the135-degree direction as shown in FIG. 21. Thus, calculating an averageof pixel values of the M pixels represented by the two coordinates mayprovide an interpolated pixel value of the 135-degreedirection-interpolated M pixel with respect to the coordinates (X, Y).Specifically, the 135-degree directional pattern analysis processingunit 64 may calculate, according to an equation (4), a 135-degreedirection-interpolated M pixel value with respect to the coordinates (X,Y) where the X and Y are both even-numbered.135-degree direction-interpolated M pixel value with respect tocoordinates (X, Y)={(M pixel value at coordinates (X−1, Y+1))+(M pixelvalue at coordinates (X+1, Y−1))}÷2  (4)where X and Y are both even-numbered

In Step S174, the 135-degree directional pattern analysis processingunit 64 performs the principle component analysis by plotting, on atwo-dimensional space of W and M, the W pixel value, together with theinterpolated M pixel value already obtained in the Step S173, withrespect to each of the pixel positions corresponding to the W pixelsalready detected within the predetermined range in the Step S172.

Specifically, within the predetermined range, the W pixel value existsin the input image at each position represented by the coordinates (X,Y), where X and Y are both even-numbered, while the 135-degreeinterpolated M pixel value at the corresponding position is alreadyobtained in the Step S173. Based on the fact, a pair of W and M pixelvalues is supposed to hold within the predetermined range with respectto more than one coordinates (X, Y), where X and Y are botheven-numbered. Thus, the 135-degree directional pattern analysisprocessing unit 64 may perform the principle component analysis byplotting more than one pair of W and M pixel values on thetwo-dimensional space of W and M.

In Step S175, the 135-degree directional pattern analysis processingunit 64 calculates the contribution rate of the first principlecomponent with respect to the principle component analysis in the StepS174 to provide the 135-degree directional pattern probability.

The contribution rate of the first principle component is found by(Variance value of the first principle component)/(Sum of variances ofeach variable), which is equivalent to (Variance value of the firstprinciple component)/(Amount of scattering of the whole samples).Specifically, for portions where there is no change of the pattern inthe 135-degree direction, the plotted pairs obtained on thetwo-dimensional space of W and M after the interpolation properlyperformed by means of the 135-degree directional interpolation ought tofall on a single straight line. In other words, when there is no changeof the pattern in the 135-degree direction, the result of the principlecomponent analysis shows that a contribution rate of the componentsother than the first principle component appears as approximately 0.Thus, the contribution rate of the first principle component isequivalent to the 135-degree directional pattern probability withrespect to the target pixel. If the principle component analysis resultsin a low contribution rate of the first principle component, it issupposed that the change of the pattern exists in the 135-degreedirection. Thus, it becomes possible to estimates that the 135-degreedirectional interpolation is not adaptable to obtain a correctlyinterpolated image.

In this processing, It should be noted that the W and M pixel values areassumed to be correlated to each other with respect to local regionssuch as those shown by H−4≦X≦H+4 and K+4≦Y≦K+4. In addition, hue usuallyremains unchanged at positions where there is the change of the pattern.That is, the W and M pixel values are usually supposed to have aproportional relation (where a coefficient of proportion is positive)with each other. Thus, in calculating the first principle component, itis also allowable to make a condition that it is necessary to passthrough the origin. Alternatively, when the direction of the firstprinciple component forms a negative inclination, it is also allowableto make an attempt to reduce a value of the 135-degree directionalpattern probability at a predetermined rate.

In Step S176, the 135-degree directional pattern analysis processingunit 64 judges whether or not the 135-degree directional patternprobability has been already calculated for all the pixels. When theresult of judgment in the Step S176 is that the 135-degree directionalpattern probability is not calculated yet for all the pixels, theprocessing returns to the Step S171, causing reprocessing to beperformed from the Step S171. When the result of judgment in the StepS176 is that the 135-degree directional pattern probability has beenalready calculated for all the pixels, the processing returns to theStep S45 in FIG. 12, and is followed by the Step S46.

The processing like the may calculate the possibility that the directionof the pattern in the vicinity of the target pixel agrees with the135-degree direction, in other words, the probability (or possibility)of having the pattern in the 135-degree direction, using the pixel valueof the pixel having the W component and the pixel value of the pixelhaving the M component among the pixels in the input image data.

The angular interpolation appropriateness calculation processing to beperformed in the Step S46 is now described with reference to theflowchart in FIGS. 22 and 23.

In this processing, the pattern probability at the respective angles aspreviously described with reference to FIGS. 14 to 21 is used as theindex to determine whether or not the angle concerned is accurate as thepattern angle, without taking any angles other than the one concernedinto consideration. On the other hand, the term of the appropriatenessis used as the index in determining whether or not the angle concernedis accurate as the pattern angle, in consideration of angles other thanthe one concerned.

In Step S201, the pattern direction determination unit 65 selects oneunprocessed target pixel.

In Step S202, the pattern direction determination unit 65 calculates,based on the 0-degree directional pattern probability and the 90-degreedirectional pattern probability of the target pixel, probability P0 thatthere is the pattern in the 0-degree direction between the 0-degreedirection and the 90-degree direction. Specifically, the patterndirection determination unit 65 determines the probability P0 that thereis the pattern in the 0-degree direction between the 0-degree directionand the 90-degree direction by the calculation of (0-degree directionalpattern probability)÷{(0-degree directional patternprobability)+(90-degree directional pattern probability)}.

In Step S203, the pattern direction determination unit 65 calculates,based on the 0-degree directional pattern probability and the 90-degreedirectional pattern probability of the target pixel, probability P90that there is the pattern in the 90-degree direction between the0-degree direction and the 90-degree direction. Specifically, thepattern direction determination unit 65 determines the probability P90that there is the pattern in the 90-degree direction between the0-degree direction and the 90-degree direction by the calculation of(90-degree directional pattern probability)÷{(0-degree directionalpattern probability)+(90-degree directional pattern probability)}.

In Step S204, the pattern direction determination unit 65 calculates,based on the 45-degree directional pattern probability and the135-degree directional pattern probability of the target pixel,probability P45 that there is the pattern in the 45-degree directionbetween the 45-degree direction and the 135-degree direction.Specifically, the pattern direction determination unit 65 determines theprobability P45 that there is the pattern in the 45-degree directionbetween the 45-degree direction and the 135-degree direction by thecalculation of (45-degree directional pattern probability)÷{(45-degreedirectional pattern probability)+(135-degree directional patternprobability)}.

In Step S205, the pattern direction determination unit 65 calculates,based on the 45-degree directional pattern probability and the135-degree directional pattern probability of the target pixel,probability P135 that there is the pattern in the 135-degree directionbetween the 45-degree direction and the 135-degree direction.Specifically, the pattern direction determination unit 65 determines theprobability P135 that there is the pattern in the 135-degree directionbetween the 45-degree direction and the 135-degree direction from thecalculation of (135-degree directional pattern probability)÷{(45-degreedirectional pattern probability)+(135-degree directional patternprobability)}.

It should be noted that the probabilities P0 and P90 obtained in theSteps S202 and S203 are assumed to be values calculated without takingthe 45-degree directional pattern probability and the 135-degreedirectional pattern probability into consideration, and that theprobabilities P45 and P135 obtained in the Steps S204 ad S205 areassumed to be values calculated without taking the 0-degree directionalpattern probability and the 90-degree directional pattern probabilityinto consideration.

In Step S206, the pattern direction determination unit 65 calculates,using the calculated probabilities P0 and P90, probability Q090 whosepattern exists in either the 0-degree direction or the 90-degreedirection. Specifically, the pattern direction determination unit 65determines the probability Q090 whose pattern exists in either the0-degree direction or the 90-degree direction by the calculation of(P0−Absolute value of P90)÷{(P0−Absolute value of P90)+(P45−Absolutevalue of P135)}.

In Step S207, the pattern direction determination unit 65 calculates,using the calculated probabilities P45 and P135, probability Q45135whose pattern exists in either the 45-degree direction or the 135-degreedirection. Specifically, the pattern direction determination unit 65determines the probability Q45135 whose pattern exists in either the45-degree direction or the 135-degree direction by the calculation of(P45−Absolute value of P135)÷{(P0−Absolute value of P90)+(P45−Absolutevalue of P135)}.

In Step S208, the pattern direction determination unit 65 calculatesappropriateness of the 0-degree directional interpolation based on thecalculated probabilities P0 and Q090. Specifically, the patterndirection determination unit 65 determines the appropriateness of the0-degree directional interpolation by the calculation of POX Q090.

In Step S209, the pattern direction determination unit 65 calculatesappropriateness of the 90-degree directional interpolation based on thecalculated probabilities P90 and Q090. Specifically, the patterndirection determination unit 65 determines the appropriateness of the90-degree directional interpolation by the calculation of POX Q090.

In Step S210, the pattern direction determination unit 65 calculatesappropriateness of the 45-degree directional interpolation based on thecalculated probabilities P45 and Q45135. Specifically, the patterndirection determination unit 65 determines the appropriateness of the45-degree directional interpolation by the calculation of P45×Q45135.

In Step S211, the pattern direction determination unit 65 calculatesappropriateness of the 135-degree directional interpolation based on thecalculated probabilities P135 and Q45135. Specifically, the patterndirection determination unit 65 determines the appropriateness of the135-degree directional interpolation by the calculation of P135×Q45135.

In Step S212, the pattern direction determination unit 65 judges whetheror not the processing has been already completed for all the pixels.When the result of judgment in the Step S212 is that the processing isnot completed yet for all the pixels, the processing returns to the StepS201, causing reprocessing to be performed from the Step S201. When theresult of judgment in the Step S212 is that the processing has beenalready completed for all the pixels, the processing returns to the StepS46 in FIG. 12, and is followed by the Step S47.

According to the processing like the above, firstly, a ratio of theappropriateness of the 0-degree directional interpolation to theappropriateness of the 90-degree directional interpolation is obtainedfrom two pattern probabilities, the 0-degree directional patternprobability and the 90-degree directional pattern probability, while aratio of the appropriateness of the 45-degree directional interpolationto the appropriateness of the 135-degree directional interpolation isobtained from two pattern probabilities, the 45-degree directionalpattern probability and the 135-degree directional pattern probability.This follows that it is estimated which of two directions orthogonal toeach other is appropriate to the interpolation to what extent.

Next, a ratio of the sum of the appropriateness of the 0-degreedirectional interpolation and the appropriateness of the 90-degreedirectional interpolation to the sum of the appropriateness of the45-degree directional interpolation and the appropriateness of the135-degree directional interpolation is obtained. Finally, theappropriateness of the 0-degree directional interpolation, theappropriateness of the 45-degree directional interpolation, theappropriateness of the 90-degree directional interpolation, and theappropriateness of the 135-degree directional interpolation areobtained.

While a difficulty is encountered in determining whether or not the twodirections are different in angle by 45 degrees, it is relatively easyto determine whether or not the two directions are different in angle by90 degrees. Specifically, it is difficult to obtain the appropriatenessof the 0-degree directional interpolation and the appropriateness of the45-degree directional interpolation respectively based on “the ratio ofthe appropriateness of the 0-degree directional interpolation to theappropriateness of the 45-degree directional interpolation”, since adifference in angle between the two directions is as slight as 45degrees. On the other hand, it is easy to obtain the appropriateness ofthe 0-degree directional interpolation and the appropriateness of the90-degree directional interpolation respectively based on “the ratio ofthe appropriateness of the 0-degree directional interpolation to theappropriateness of the 90-degree directional interpolation”, since adifference in angle between the two directions is large. Thus, theprocedure like the enables the appropriateness of the interpolation inthe directions at the respective angles to be obtained without needingany complicated operational processing.

One specific instance is now described on what type of theappropriateness is obtained with respect to each angle when the patternexists in the 0-degree direction.

Since there is the pattern in the 0-degree direction, “the 0-degreedirectional pattern probability” is high anyway, whereas “the patternprobability in the 90-degree direction” orthogonal to the 0-degreedirection is low. For that reason, the probabilities P0 and P90calculated in the Steps S201 and 202 respectively appear as P0≈1 andP90≈0. Then, with respect to the 45-degree direction and the 135-degreedirection, these directions are different from the actual patterndirection in angle by the same degrees (or 45 degrees), so that “the45-degree directional pattern probability≈the 135-degree directionalpattern probability” holds true. For that reason, the relation betweenthe probabilities P45 and P135 calculated in the Steps S204 and S205results in P45≈P135.

Thus, in the Step S206, “the probability Q90≈1÷(1+0)≈1” is obtained, andin the Step S207, “the probability Q45135≈0÷(1+0)≈0” is obtained. Then,in the Step S208, “the appropriateness of the 0-degree directionalinterpolation≈1×1≈1” is obtained, and in the Step S209, “theappropriateness of the 90-degree directional interpolation≈0” isobtained. Further, in the Step S210, “the appropriateness of the45-degree directional interpolation≈0” is obtained, and in the StepS211, “the appropriateness of the 135-degree directionalinterpolation≈0” is obtained. Thus, the interpolation processing in the0-degree direction is adapted to perform the demosaic processing,permitting a satisfactory image to be obtained.

By the way, with respect to a projected image portion of a subjecthaving no pattern, the 0-degree directional pattern probability, the45-degree directional pattern probability, the 90-degree directionalpattern probability and the 135-degree directional pattern probabilityare all supposed to take large values equivalently, and as a result,“the appropriateness of the 0-degree directional interpolation≈0.25”,“the appropriateness of the 45-degree directional interpolation≈0.25”,“the appropriateness of the 90-degree directional interpolation≈0.25”and “the appropriateness of the 135-degree directionalinterpolation≈0.25” hold true respectively. Accordingly, the same degreeof weights is applied to the interpolation processing in all thedirections.

The 0-degree direction-interpolated G component image generationprocessing to be performed in the Step S47 in FIG. 13 is now describedwith reference to the flowchart in FIG. 24.

With respect to the filter array of the color filter 22 previouslydescribed with reference to FIG. 6, the G values exist in every otherpixel in the input image on lines whose Y-axis values are odd-numbered,so that the 0-degree directional interpolation is adaptable to calculatethe interpolated pixel value of G for the pixels indicated by M.However, there is no G value in the input image on lines whose Y-axisvalues are even-numbered, so that the 0-degree directional interpolationis not adaptable to directly calculate the interpolated pixel value ofG. Thus, an attempt is made to firstly calculate a W component imagewith respect to all the pixels based on the 0-degree directionalinterpolation, followed by performing the calculation of the G valuewith respect to all the pixels based on the correlation between the Gand W pixel values.

First, in Step S241, the 0-degree direction-interpolated G componentimage calculation processing unit 71 selects one target pixel thatremains unprocessed with respect to extraction or interpolation of the Wvalue.

In Step S242, the 0-degree direction-interpolated G component imagecalculation processing unit 71 judges whether or not the target pixel isthe pixel having the W pixel value. The pixels indicated by W areoriginally existent at positions represented by the coordinates (X, Y)where X and Y are both even-numbered. Thus, when the result of judgmentin the Step S242 is that the target pixel is the pixel having the Wpixel value, it suffices to assume this pixel value to be a W value withrespect to the coordinates (X, Y), and the processing goes on to StepS251 described later.

When the result of judgment in the Step S242 is that the target pixel isnot the pixel having the W pixel value, the 0-degreedirection-interpolated G component image calculation processing unit 71judges whether or not the W pixel exists as the pixel adjacent in the0-degree direction to the target pixel in Step S243. The pixels at thecoordinates (X−1, Y) and (X+1, Y) adjacent in the 0-degree direction tothe pixels at the coordinates (X, Y) where X is odd-numbered and Y iseven-numbered hold the W pixel value. Thus, when the result of judgmentin the Step S243 is that no W pixel exists in the 0-degree directionwith respect to the target pixel, the processing goes on to Step S245described later.

When the result of judgment in the Step S243 is that the W pixel existsin the 0-degree direction with respect to the target pixel, the 0-degreedirection-interpolated G component image calculation processing unit 71obtains the interpolated pixel value of W based on the pixel value ofthe W pixel adjacent in the 0-degree direction in Step S244.Specifically, the 0-degree direction-interpolated G component imagecalculation processing unit 71 calculates an average of the pixel valuewith respect to the coordinates (X−1, Y) and the pixel value withrespect to the coordinates (X+1, Y) to provide the interpolated pixelvalue of W with respect to the coordinates (X, Y), and the processinggoes on to Step S251 described later.

When the result of judgment in the Step S243 is that no W pixel existsin the 0-degree direction with respect to the target pixel, the 0-degreedirection-interpolated G component image calculation processing unit 71judges whether or not the target pixel is the pixel having the G pixelvalue in the Step S245. When the result of judgment in the Step S245 isthat the target pixel is not the pixel having the G pixel value, theprocessing goes on to Step S248 described later.

When the result of judgment in the Step S245 is that the target pixel isthe pixel having the G pixel value, the 0-degree direction-interpolatedG component image calculation processing unit 71 obtains theinterpolated pixel value of M with respect to the target pixel based oneach pixel value of the M pixels adjacent in the 0-degree direction inStep S246. Specifically, the pixels indicated by G exist at positionsrepresented by the coordinates (X, Y) where X is even-numbered and Y isodd-numbered. Then, the pixels indicated by M exist as the pixelsadjacent in the 0-degree direction to each G pixel, that is, atpositions represented by the coordinates (X−1, Y) and (X+1, Y), so thatit is possible to assume the average of the pixel values with respect tothe two coordinates to be the interpolated pixel value of M with respectto the coordinates (X, Y).

In Step S247, the 0-degree direction-interpolated G component imagecalculation processing unit 71 calculates the W pixel value by addingthe G pixel value and the interpolated pixel value of M with respect tothe target pixel based on that the equation of “W pixel value=G pixelvalue+M pixel value” holds true, and the processing goes on to Step S251described later.

When the result of judgment in the Step S245 is that the target pixel isnot the pixel having the G pixel value, the 0-degreedirection-interpolated G component image calculation processing 71judges whether or not the target pixel is the pixel having the M pixelvalue in Step S248. When the result of judgment in the Step S248 is thatthe target pixel is not the pixel having the M pixel value, theprocessing goes on to the Step S251 described later.

When the result of judgment in the Step S248 is that the target pixel isthe pixel having the M pixel value, the 0-degree direction-interpolatedG component image calculation processing unit 71 obtains theinterpolated pixel value of G with respect to the target pixel based onthe G pixel values the pixels adjacent in the 0-degree direction to thetarget pixel hold in Step S249. Specifically, the pixels indicated by Mexist at positions represented by the coordinates (X, Y) where X and Yare both odd-numbered. Then, the pixels indicated by G exist as thepixels adjacent in the 0-degree direction to each M pixel, that is, atpositions represented by the coordinates (X−1, Y) and (X+1, Y), so thatit is possible to assume the average of the pixel values with respect tothe two coordinates to be the interpolated pixel value of G with respectto the coordinates (X, Y).

In Step S250, the 0-degree direction-interpolated G component imagecalculation processing unit 71 calculates the W pixel value by addingthe M pixel value and the interpolated pixel value of G with respect tothe target pixel based on that the equation of “W pixel value=G pixelvalue+M pixel value” holds true, and the processing goes on to the StepS251 described later.

When the result of judgment in the Step S242 is that the target pixel isthe pixel having the W pixel value, or after the processing of the StepS244 or 247, or when the result of judgment in the Step S248 is that thetarget pixel is not the pixel having the M pixel value, or after theprocessing of the Step S250, the 0-degree direction-interpolated Gcomponent image calculation processing unit 71 judges whether or not theW value has been already obtained for all the pixels in the Step S251.The result of judgment in the Step S251 is that the W value is notobtained yet for all the pixels, the processing returns to the StepS241, causing reprocessing to be performed from the Step S241.

When the result of judgment in the Step S251 is that the W value hasbeen already obtained for all the pixels, the 0-degreedirection-interpolated G component image calculation processing unit 71selects one target pixel that remains unprocessed with respect todetermination of the G value in Step S252.

In Step S253, the 0-degree direction-interpolated G component imagecalculation processing unit 71 detects the G pixels existing within thepredetermined range from the target pixel, wherein the predeterminedrange indicates a range experientially of experimentally determinedbased on the conditions such as the number of samples used for theprinciple component analysis and the closeness of correlation dependingon a distance from the target pixel. Assuming that the target pixelposition is represented as the coordinates (H, K), for instance, thepredetermined range may include those such as a range having a size of9×9 pixels represented by the coordinates (X, Y) where H−4≦X≦H+4 andK+4≦Y≦K+4. Alternatively a range having a different size or a rangecovering the predetermined number of pixels selected from the pixelsexisting close to each other with the target pixel as the center mayalso be used.

The predetermined range stated herein and a predetermined range withrespect to the 90-degree direction-interpolated G component imagegeneration processing as described later may be the same as or differentfrom the predetermined range applied to the 0-degree, 45-degree,90-degree or 135-degree directional pattern probability calculationprocessing.

In Step S254, the 0-degree direction-interpolated G component imagecalculation processing unit 71 performs the principle component analysisafter plotting, on the two-dimensional space of G and W, the G pixelvalue of each pixel previously detected in the Step S253, together withthe W value previously obtained with respect to the detected G pixelposition.

Specifically, since the W value with respect to all the pixel positionshas been already obtained through the processing, it is proper thatthere is also provided the W pixel value with respect to the detected Gpixel positions. Thus, assuming that the predetermined range is definedas H−4≦X≦H+4 and K+4≦Y≦K+4 with respect to the coordinates (X, Y) of thetarget pixel, the G pixel represented by the coordinates (X, Y) where Xis even-numbered and Y is odd-numbered exists at more than one positionwithin the predetermined range, so that it is possible to form more thanone pair of W and G pixel values. By plotting the more than one pair ofW and G pixel values on the two-dimensional space of W and G to performthe principle component analysis, a straight line of the first principlecomponent may be obtained as shown in FIG. 25.

In this processing, when it is assumed that the variance and the averageof the detected G pixel values within the predetermined range in theStep S253 are respectively represented by varianceG and AveG, and thatthe variance and the average of the W pixel values with respect to thesedetected G pixel positions are respectively represented as varianceW andAveW, an equation of the straight line of the first principle componentis shown by the following equation (5).G=S×√{square root over (variance G)}÷√{square root over (varianceW)}×(W−AveW)+AveG  (5)where a coefficient S is assumed to be +1 when the correlation ispositive, and to be −1 when the correlation is negative.

In Step S255, the 0-degree direction-interpolated G image calculationprocessing unit 71 calculates the G pixel value based on the straightline resulting from the principle component analysis, together with theW pixel value with respect to the target pixel.

Specifically, it suffices to obtain the G pixel value corresponding tothe W pixel value with respect to the target pixel on the straight lineof the first principle component, as shown in FIG. 25, and thus, the Gvalue is calculated by substituting the W value with respect to thetarget pixel (H, K) in the equation of the straight line as shown by theequation (5), wherein the W and G values are assumed to be correlated toeach other within the predetermined range such as those where H−4≦X≦H+4and K+4≦Y≦K+4, for instance.

In Step S256, the 0-degree direction-interpolated G image calculationprocessing unit 71 judges whether or not the G value has been alreadyobtained for all the pixels. When the result of judgment in the StepS256 is that the G value is not obtained yet for all the pixels, theprocessing returns to the Step S252, causing reprocessing to beperformed from the Step S252. When the result of judgment in the StepS256 is that the G value has been already obtained for all the pixels,the processing returns to the Step S47 in FIG. 13, and is followed bythe Step S48.

The processing like the may provide the G component image based on the0-degree directional interpolation by, after calculating the W componentimage with respect to all the pixels through the 0-degree directionalinterpolation, performing the calculation of the G value with respect toall the pixels based on the correlation between the G and W values usingthe result of the calculation of the W component image.

The 45-degree direction-interpolated G component image generationprocessing to be performed in the Step S48 in FIG. 13 is now describedwith reference to the flowchart in FIG. 26.

With respect to the filter array of the color filter 22 previouslydescribed with reference to FIG. 6, the G values exist in every otherpixel in the input image on 45-degree directional lines made up of thecoordinates (X, Y) where X+Y is odd-numbered, so that the 45-degreedirectional interpolation is adaptable to calculate the interpolatedpixel value of G for the pixels indicated by R or B. While there is no Gvalue in the input image on 45-degree directional lines made up of thecoordinates (X, Y) where X+Y is even-numbered, the 45-degree directionallines consist of the W or M pixels, so that it is possible to obtain theG value by the calculation of W−M.

First, in Step S281, the 45-degree direction-interpolated G componentimage calculation processing unit 72 selects one notice pixel thatremains unprocessed with respect to extraction or interpolation of the Gvalue.

In Step S282, the 45-degree direction-interpolated G component imagecalculation processing unit 72 judges whether or not the target pixel isthe pixel having the G pixel value. The pixels indicated by G areoriginally existent at positions represented by the coordinates (X, Y)where X is even-numbered and Y is odd-numbered. Thus, when the result ofjudgment in the Step S282 is that the target pixel is the pixel havingthe G pixel value, it suffices to assume this pixel value to be a Gvalue with respect to the coordinates (X, Y), and the processing goes onto Step S291 described later.

When the result of judgment in the Step S282 is that the target pixel isnot the pixel having the G pixel value, the 45-degreedirection-interpolated G component image calculation processing unit 72judges whether or not the G pixel exists as the pixel adjacent in the45-degree direction with respect to the target pixel in Step S283. Thepixels at the coordinates (X−1, Y−1) and (X+1, Y+1) adjacent in the45-degree direction to the pixel at the coordinates (X, Y) where X+Y isodd-numbered hold the G pixel value. Thus, when the result of judgmentin the Step S283 is that no G pixel exists in the 45-degree directionwith respect to the target pixel, the processing goes on to Step S285described later.

When the result of judgment in the Step S283 is that the G pixel existsin the 45-degree direction with respect to the target pixel, the45-degree direction-interpolated G component image calculationprocessing unit 72 obtains the interpolated pixel value of G based onthe pixel value of the G pixel in the 45-degree direction in Step S284.Specifically, the 45-degree direction-interpolated G component imagecalculation processing unit 72 calculates an average of the pixel valuewith respect to the coordinates (X−1, Y−1) and the pixel value withrespect to the coordinates (X+1, Y+1) to provide the interpolated pixelvalue of G with respect to the coordinates (X, Y), and the processinggoes on to Step S291 described later.

When the result of judgment in the Step S283 is that no G pixel existsin the 45-degree direction with respect to the target pixel, the45-degree direction-interpolated G component image calculationprocessing unit 72 judges whether or not the target pixel is the pixelhaving the W pixel value in the Step S285. When the result of judgmentin the Step S285 is that the target pixel is not the pixel having the Wpixel value, the processing goes on to Step S288 described later.

When the result of judgment in the Step S285 is that the target pixel isthe pixel having the W pixel value, the 45-degree direction-interpolatedG component image calculation processing unit 72 obtains theinterpolated pixel value of M based on the pixel value of the M pixel inthe 45-degree direction in Step S286. Specifically, the pixels indicatedby W exist at positions represented by the coordinates (X, Y) where Xand Y are both even-numbered. Then, the pixels indicated by M exist asthe pixels adjacent in the 45-degree direction to each W pixel, that is,at positions represented by the coordinates (X−1, Y−1) and (X+1, Y+1),so that it is possible to assume the average of the pixel values withrespect to the two coordinates to be the interpolated pixel value of Mwith respect to the coordinates (X, Y).

In Step S287, the 45-degree direction-interpolated G component imagecalculation processing unit 72 calculates the G pixel value bysubtracting the interpolated pixel value of M from the W pixel valuewith respect to the target pixel based on that the equation of “G pixelvalue=W pixel value−M pixel value” holds true, and the processing goeson to Step S291 described later.

When the result of judgment in the Step S285 is that the target pixel isnot the pixel having the W pixel value, the 45-degreedirection-interpolated G component image calculation processing 72judges whether or not the target pixel is the pixel having the M pixelvalue in Step S288. When the result of judgment in the Step S288 is thatthe target pixel is not the pixel having the M pixel value, theprocessing goes on to the Step S291 described later.

When the result of judgment in the Step S288 is that the target pixel isthe pixel having the M pixel value, the 45-degree direction-interpolatedG component image calculation processing unit 72 obtains theinterpolated pixel value of W based on the W pixel values the pixelsadjacent in the 45-degree direction to the target pixel hold in StepS289. Specifically, the pixels indicated by M exist at positionsrepresented by the coordinates (X, Y) where X and Y are bothodd-numbered. Then, the pixels indicated by W exist as the pixelsadjacent in the 45-degree direction to each M pixel, that is, atpositions represented by the coordinates (X−1, Y−1) and (X+1, Y+1), sothat it is possible to assume the average of the pixel values withrespect to the two coordinates to be the interpolated pixel value of Wwith respect to the coordinates (X, Y).

In Step S290, the 45-degree direction-interpolated G component imagecalculation processing unit 72 calculates the G pixel value bysubtracting the M pixel value from the interpolated pixel value of Wwith respect to the target pixel based on that the equation of “G pixelvalue=W pixel value−M pixel value” holds true, and the processing goeson to the Step S291 described later.

When the result of judgment in the Step S282 is that the target pixel isthe pixel having the G pixel value, or after the processing of the StepS284 or 287, or when the result of judgment in the Step S288 is that thetarget pixel is not the pixel having the M pixel value, or after theprocessing of the Step S290, the 45-degree direction-interpolated Gcomponent image calculation processing unit 72 judges whether or not theG value has been already obtained for all the pixels in the Step S291.When the result of judgment in the Step S291 is that the G value is notobtained yet for all the pixels, the processing returns to the StepS281, causing reprocessing to be performed from the Step S281. When theresult of judgment in the Step S291 is that the G value has been alreadyobtained for all the pixels, the processing returns to the Step S48 inFIG. 13, and is followed by the Step S49.

The processing like the may provide the G component image based on the45-degree directional interpolation.

The 90-degree direction-interpolated G component image generationprocessing to be performed in the Step S49 in FIG. 13 is now describedwith reference to the flowchart in FIG. 27.

With respect to the filter array of the color filter 22 previouslydescribed with reference to FIG. 6, the G values exist in every otherpixel in the input image on lines whose X-axis values are even-numbered,so that the 90-degree directional interpolation is adaptable tocalculate the interpolated pixel value of G for the pixels indicated byW. However, there is no G value in the input image on lines whose X-axisvalues are odd-numbered, so that the 90-degree directional interpolationis not adaptable to directly calculate the interpolated pixel value ofG. Thus, an attempt is made to firstly calculate the M component imagewith respect to all the pixels based on the 90-degree directionalinterpolation, followed by performing the calculation of the G valuewith respect to all the pixels based on the correlation between the Gand M values.

First, in Step S321, the 90-degree direction-interpolated G componentimage calculation processing unit 73 selects one notice pixel thatremains unprocessed with respect to extraction or interpolation of the Mvalue.

In Step S322, the 90-degree direction-interpolated G component imagecalculation processing unit 73 judges whether or not the target pixel isthe pixel having the M pixel value. The pixels indicated by M areoriginally existent at positions represented by the coordinates (X, Y)where X and Y are both odd-numbered. Thus, when the result of judgmentin the Step S322 is that the target pixel is the pixel having the Mpixel value, it suffices to assume this pixel value to be the M valuewith respect to the coordinates (X, Y), and the processing goes on toStep S331 described later.

When the result of judgment in the Step S322 is that the target pixel isnot the pixel having the M pixel value, the 90-degreedirection-interpolated G component image calculation processing unit 73judges whether or not the M pixel exists as the pixel adjacent in the90-degree direction with respect to the target pixel in Step S323. Thepixels at the coordinates (X, Y−1) and (X, Y+1) adjacent in the90-degree direction to the pixel at the coordinates (X, Y) where X isodd-numbered and Y is even-numbered hold the M pixel value. Thus, whenthe result of judgment in the Step S323 is that no M pixel exists in the90-degree direction with respect to the target pixel, the processinggoes on to Step S325 described later.

When the result of judgment in the Step S323 is that the M pixel existsin the 90-degree direction with respect to the target pixel, the90-degree direction-interpolated G component image calculationprocessing unit 73 obtains the interpolated pixel value of M based onthe pixel value of the M Pixel in the 90-degree direction in Step S324.Specifically, the 90-degree direction-interpolated G component imagecalculation processing unit 73 calculates an average of the pixel valuewith respect to the coordinates (X, Y−1) and the pixel value withrespect to the coordinates (X, Y+1) to provide the interpolated pixelvalue of M with respect to the coordinates (X, Y), and the processinggoes on to Step S331 described later.

When the result of judgment in the Step S323 is that no M pixel existsin the 90-degree direction with respect to the target pixel, the90-degree direction-interpolated G component image calculationprocessing unit 73 judges whether or not the target pixel is the pixelhaving the G pixel value in Step S325. When the result of judgment inthe Step S325 is that the target pixel is not the pixel having the Gpixel value, the processing goes on to Step S328 described later.

When the result of judgment in the Step S325 is that the target pixel isthe pixel having the G pixel value, the 90-degree direction-interpolatedG component image calculation processing unit 73 obtains theinterpolated pixel value of W based on the pixel value of the W pixel inthe 90-degree direction in Step S326. Specifically, the pixels indicatedby G exist at positions represented by the coordinates (X, Y) where X iseven-numbered and Y is odd-numbered. Then, the pixels indicated by Wexist as the pixels adjacent in the 90-degree direction to each G pixel,that is, at positions represented by the coordinates (X, Y−1) and (X,Y+1), so that it is possible to assume the average of the pixel valueswith respect to the two coordinates to be the interpolated pixel valueof W with respect to the coordinates (X, Y).

In Step S327, the 90-degree direction-interpolated G component imagecalculation processing unit 73 calculates the M pixel value bysubtracting the G pixel value from the interpolated pixel value of Wwith respect to the target pixel based on that the equation of “M pixelvalue=W pixel value−G pixel value” holds true, and the processing goeson to Step S331 described later.

When the result of judgment in the Step S325 is that the target pixel isnot the pixel having the G pixel value, the 90-degreedirection-interpolated G component image calculation processing unit 73judges whether or not the target pixel is the pixel having the W pixelvalue in Step S328. When the result of judgment in the Step S328 is thatthe target pixel is not the pixel having the W pixel value, theprocessing goes on to the Step S331 described later.

When the result of judgment in the Step S328 is that the target pixel isthe pixel having the W pixel value, the 90-degree direction-interpolatedG component image calculation processing unit 73 obtains theinterpolated pixel value of G based on the G pixel values the pixelsadjacent in the 90-degree direction with respect to the target pixelhold in Step S329. Specifically, the pixels indicated by W exist atpositions represented by the coordinates (X, Y) where X and Y are botheven-numbered. Then, the pixels indicated by G exist as the pixelsadjacent in the 90-degree direction to each W pixel, that is, atpositions represented by the coordinates (X, Y−1) and (X, Y+1), so thatit is possible to assume the average of the pixel values with respect tothe two coordinates to be the interpolated pixel value of G with respectto the coordinates (X, Y).

In Step S330, the 90-degree direction-interpolated G component imagecalculation processing unit 73 calculates the M pixel value bysubtracting the interpolated pixel value of G from the W pixel valuewith respect to the target pixel based on that the equation of “M pixelvalue=W pixel value−G pixel value” holds true, and the processing goeson to the Step S331 described later.

When the result of judgment in the Step S322 is that the target pixel isthe pixel having the M pixel value, or after the processing of the StepS324 or 327, or when the result of judgment in the Step S328 is that thetarget pixel is not the pixel having the W pixel value, or after theprocessing of the Step S330, the 90-degree direction-interpolated Gcomponent image calculation processing unit 73 judges whether or not theM value has been already obtained for all the pixels in the Step S331.The result of judgment in the Step S331 is that the M value is notobtained yet for all the pixels, the processing returns to the StepS321, causing reprocessing to be performed from the Step S321.

When the result of judgment in the Step S331 is that the M value hasbeen already obtained for all the pixels, the 90-degreedirection-interpolated G component image calculation processing unit 73selects one notice pixel that remains unprocessed with respect todetermination of the G value in Step S332.

In Step S333, the 90-degree direction-interpolated G component imagecalculation processing unit 73 detects the G pixels existing within thepredetermined range from the target pixel, wherein the predeterminedrange indicates a range experientially or experimentally determinedbased on the conditions such as the number of samples used for theprinciple component analysis and the closeness of correlation dependingon a distance from the target pixel. Assuming that the target pixelposition is represented as the coordinates (H, K), for instance, thepredetermined range may include those such as a range having a size of9×9 pixels represented by the coordinates (X, Y) where H−4≦X≦H+4 andK+4≦Y≦K+4. Alternatively, a range having a different size or a rangecovering the predetermined number of pixels selected from the pixelsexisting close to each other with the target pixel as the center mayalso be used.

In Step S334, the 90-degree direction-interpolated G component imagecalculation processing unit 73 performs the principle component analysisafter plotting, on the two-dimensional space of G and M, the G pixelvalue of each pixel already detected in the Step S333, together with theM value already obtained with respect to the detected G pixel position.

Specifically, since the M value with respect to all the pixel positionshas been already obtained through the processing, it is proper thatthere is also provided the M pixel value with respect to the detected Gpixel positions. Thus, assuming that the predetermined range is definedas H−4≦X≦H+4 and K+4≦Y≦K+4 with respect to the coordinates (H, K) of thetarget pixel, the G pixel represented by the coordinates (X, Y), where Xis even-numbered and Y is odd-numbered, exists at more than one positionwithin the predetermined range, so that it is possible to form more thanone pair of M and G values. By plotting the more than one pair of M andG values on the two-dimensional space of M and G to perform theprinciple component analysis, a straight line of the first principlecomponent may be obtained as shown in FIG. 28.

In this processing, when it is assumed that the variance and the averageof the detected G pixel values within the predetermined range in theStep S333 are respectively represented as varianceG and AveG, and thatthe variance and the average of the M pixel values with respect to thesedetected G pixel positions are respectively represented as variances andAveM, an equation of the straight line of the first principle componentis shown by the following equation (6).G=S×√{square root over (variance G)}÷√{square root over (varianceM)}×(M−AveM)+AveG  (6)where a coefficient S is assumed to be +1 when the correlation ispositive, and to be −1 when the correlation is negative.

In Step S335, the 90-degree directional interpolated G image calculationprocessing unit 73 calculates the G pixel value based on the straightline resulting from the principle component analysis, together with theM pixel value with respect to the target pixel.

Specifically, it suffices to obtain the G pixel value corresponding tothe M pixel value of the target pixel with respect to the pixels on thestraight line of the first principle component, as shown in FIG. 28, andthus, the G value is calculated by substituting the M value with respectto the target pixel (H, K) in the equation of the straight line as shownby the equation (6), wherein the M and G values are assumed to becorrelated to each other within the predetermined range such as thosewhere H−4≦X≦H+4 and K+4≦Y≦K+4, for instance.

In Step S336, the 90-degree direction-interpolated G image calculationprocessing unit 73 judges whether or not the G value has been obtainedfor all the pixels. When the result of judgment in the Step S336 is thatthe G value is not obtained yet for all the pixels, the processingreturns to the Step S332, causing reprocessing to be performed from theStep S332. When the result of judgment in the Step S336 is that the Gvalue has been already obtained for all the pixels, the processingreturns to the Step S49 in FIG. 13, and is followed by the Step S50.

The processing like the may provide the G component image based on the90-degree directional interpolation by, after calculating the Mcomponent image with respect to all the pixels through the 90-degreedirectional interpolation, performing the calculation of the G valuewith respect to all the pixels based on the correlation between the Gand M values using the result of the calculation of the M componentimage.

The 135-degree direction-interpolated G component image generationprocessing to be performed in the Step S50 in FIG. 13 is now describedwith reference to the flowchart in FIG. 29.

With respect to the filter array of the color filter 22 previouslydescribed with reference to FIG. 6, the G values exist in every otherpixel in the input image on 135-degree directional lines made up of thecoordinates (X, Y) where X−Y is odd-numbered, so that the 135-degreedirectional interpolation is adaptable to calculate the interpolatedpixel value of G for the pixels indicated by R or B. While there is no Gvalue in the input image on 135-degree directional lines made up of thecoordinates (X, Y) where X−Y is even-numbered, the 135-degreedirectional lines consist of the W or M pixels, so that it is possibleto obtain the G value by the calculation of W−M.

First, in Step S361, the 135-degree direction-interpolated G componentimage calculation processing unit 74 selects one notice pixel thatremains unprocessed with respect to extraction or interpolation of the Gvalue.

In Step S362, the 135-degree direction-interpolated G component imagecalculation processing unit 74 judges whether or not the target pixel isthe pixel having the G pixel value. The pixels indicated by G areoriginally existent at positions represented by the coordinates (X, Y)where X is even-numbered and Y is odd-numbered. Thus, when the result ofjudgment in the Step S362 is that the target pixel is the pixel havingthe G pixel value, it suffices to assume this pixel value to be the Gvalue with respect to the coordinates (X, Y), and the processing goes onto Step S371 described later.

When the result of judgment in the Step S362 is that the target pixel isnot the pixel having the G pixel value, the 135-degreedirection-interpolated G component image calculation processing unit 74judges whether or not the G pixel exists as the pixel adjacent in the135-degree direction with respect to the target pixel in Step S363. Thepixels at the coordinates (X−1, Y+1) and (X+1, Y−1) adjacent in the135-degree direction to the pixel at the coordinates (X, Y) where X+Y isodd-numbered hold the G pixel value. Thus, when the result of judgmentin the Step S363 is that no G pixel exists in the 135-degree directionwith respect to the target pixel, the processing goes on to Step S365described later.

When the result of judgment in the Step S363 is that the G pixel existsin the 135-degree direction with respect to the target pixel, the135-degree direction-interpolated G component image calculationprocessing unit 74 obtains the interpolated pixel value of G based onthe pixel value of the G pixel in the 135-degree direction in Step S364.Specifically, the 135-degree direction-interpolated G component imagecalculation processing unit 74 calculates an average of the pixel valuewith respect to the coordinates (X−1, Y+1) and the pixel value withrespect to the coordinates (X+1, Y−1) to provide the interpolated pixelvalue of G with respect to the coordinates (X, Y), and the processinggoes on to Step S371 described later.

When the result of judgment in the Step S363 is that no G pixel existsin the 135-degree direction with respect to the target pixel, the135-degree direction-interpolated G component image calculationprocessing unit 74 judges whether or not the target pixel is the pixelhaving the W pixel value in Step S365. When the result of judgment inthe Step S365 is that the target pixel is not the pixel having the Wpixel value, the processing goes on to Step S368 described later.

When the result of judgment in the Step S365 is that the target pixel isthe pixel having the W pixel value, the 135-degreedirection-interpolated G component image calculation processing unit 74obtains the interpolated pixel value of M based on the pixel value ofthe M pixel in the 135-degree direction in Step S366. Specifically, thepixels indicated by W exist at positions represented by the coordinates(X, Y) where X and Y are both even-numbered. Then, the pixels indicatedby M exist as the pixels adjacent in the 135-degree direction to each Wpixel, that is, at positions represented by the coordinates (X−1, Y+1)and (X+1, Y−1), so that it is possible to assume the average of thepixel values with respect to the two coordinates to be the interpolatedpixel value of M with respect to the coordinates (X, Y).

In Step S367, the 135-degree direction-interpolated G component imagecalculation processing unit 74 calculates the G pixel value bysubtracting the interpolated pixel value of M from the W pixel valuewith respect to the target pixel based on that the equation of “G pixelvalue=W pixel value−M pixel value” holds true, and the processing goeson to Step S371 described later.

When the result of judgment in the Step S365 is that the target pixel isnot the pixel having the W pixel value, the 135-degreedirection-interpolated G component image calculation processing 74judges whether or not the target pixel is the pixel having the M pixelvalue in Step S368. When the result of judgment in the Step S368 is thatthe target pixel is not the pixel having the M pixel value, theprocessing goes on to the Step S371 described later.

When the result of judgment in the Step S368 is that the target pixel isthe pixel having the M pixel value, the 135-degreedirection-interpolated G component image calculation processing unit 74obtains the interpolated pixel value of W based on the W pixel valuesthe pixels adjacent in the 135-degree direction with respect to thetarget pixel hold in Step S369. Specifically, the pixels indicated by Mexist at positions represented by the coordinates (X, Y) where X and Yare both odd-numbered. Then, the pixels indicated by W exist as thepixels adjacent in the 135-degree direction to each M pixel, that is, atpositions represented by the coordinates (X−1, Y+1) and (X+1, Y−1), sothat it is possible to assume the average of the pixel values withrespect to the two coordinates to be the interpolated pixel value of Wwith respect to the coordinates (X, Y).

In Step S370, the 135-degree direction-interpolated G component imagecalculation processing unit 74 obtains the G pixel value by subtractingthe M pixel value from the interpolated pixel value of W with respect tothe target pixel based on that the equation of “G pixel value=W pixelvaue−M pixel value” holds true, and the processing goes on to the StepS371 described later.

When the result of judgment in the Step S362 is that the target pixel isthe pixel having the G pixel value, or after the processing of the StepS364 or 367, or when the result of judgment in the Step S368 is that thetarget pixel is not the pixel having the M pixel value, or after theprocessing of the Step S370, the 135-degree direction-interpolated Gcomponent image calculation processing unit 74 judges whether or not theG value has been already obtained for all the pixels in Step S371.

When the result of judgment in the Step S371 is that the G value is notobtained yet for all the pixels, the processing returns to the StepS361, causing reprocessing to be performed from the Step S361. When theresult of judgment in the Step S371 is that the G value has been alreadyobtained for all the pixels, the processing returns to the Step S50 inFIG. 13 and moves onto the Step S51.

The processing like the may provide the G component image based on the135-degree directional interpolation.

Then, as described the above, in the Step S51 in FIG. 13, the Gcomponent image to be generated under consideration of the patterndirection and by the use of the property that the equation of W−M=Gholds for the W and M component pixels is obtained by, after multiplyingthe G component image already corrected in the direction at each angleby the probability correction appropriateness as the weighting factor,performing the addition of the resultant G component image for eachpixel.

The R component pixel value calculation processing to be performed inthe Step S52 in FIG. 13 is now described with reference to the flowchartin FIG. 30.

In Step S401, the R component image calculation processing unit 81selects one unprocessed target pixel.

In Step S402, the R component image calculation processing unit 81detects the R pixels existing within the predetermined range from thetarget pixel, wherein the predetermined range indicates a rangeexperientially or experimentally determined based on the conditions suchas the number of samples used for the principle component analysis andthe closeness of correlation depending on a distance from the targetpixel. Assuming that the target pixel position is represented as thecoordinates (H, K), for instance, the predetermined range may includethose such as a range having a size of 9×9 pixels represented by thecoordinates (X, Y), where H−4≦X≦H+4 and K+4≦Y≦K+4. Alternatively, arange having a different size or a range covering the predeterminednumber of pixels selected from the pixels existing close to each otherwith the target pixel as the center may also be used.

Further, the predetermined range stated herein and a predetermined rangewith respect to the B component pixel value calculation processingdescribed later may be the same as or different from the predeterminedrange applied to any one of the processing.

Assuming that the target pixel position is represented as (H, K), forinstance, the pixels indicated by R exist at positions represented bythe coordinates (X, Y) where H−4≦X≦H+4, K+4≦Y≦K+4, X is odd-numbered, Yis even-numbered and X−Y−3 takes a multiple of 4.

In Step S403, the R component image calculation processing unit 81acquires the G pixel value estimated through the processing in the StepS51 in FIG. 13 with respect to all the detected R pixel (X, Y)positions. Specifically, the R component image calculation processingunit 81 acquires more than one pair of R and G pixel values with respectto all the detected R pixel (X, Y) positions.

In Step S404, the R component image calculation processing unit 81performs the principle component analysis after plotting the pairs of Rand G pixel values on the two-dimensional space of R and G, as shown inFIG. 31.

In this processing, when it is assumed that the variance and the averageof the R pixel values within the predetermined range are respectivelyrepresented as varianceR and AveR, and that the variance and the averageof the G pixel values with respect to these R pixel positions arerespectively represented as varianceG and AveG, an equation of astraight line of the first principle component is shown as the followingequation (7).R=S×√{square root over (variance R)}÷√{square root over (varianceG)}×(G−AveG)+AveR  (7)where a coefficient S is assumed to be +1 when the correlation ispositive, and to be −1 when the correlation is negative.

In Step S405, the R component image calculation processing unit 81obtains the R value based on the straight line resulting from theprinciple component analysis, together with the G value with respect tothe target pixel. Specifically, the R component image calculationprocessing unit 81 calculates the R value by substituting the G pixelvalue with respect to the target pixel in the equation (7)

In Step S406, the R component image calculation processing unit 81judges whether or not the R value has been already obtained for all thepixels. When the result of judgment in the Step S406 is that the R valueis not obtained yet for all the pixels, the processing returns to theStep S401, causing reprocessing to be performed from the Step S401. Whenthe result of judgment in the Step S406 is that the R value has beenalready obtained for all the pixels, the processing returns to the StepS52 in FIG. 13, and is followed by the Step S53.

The processing like the may provide the R component pixel value for allthe pixels based on the pixel value of the R component contained in theimage data, together with the G pixel value estimated through theprocessing of the Step S51 in FIG. 13.

In this processing, It should be noted that the processing is put intopractice assuming that the R and G pixel values are correlated to eachother within the predetermined range such as those where H−4≦X≦H+4 andK+4≦Y≦K+4, for instance.

The B component pixel value calculation processing to be performed inthe Step S53 in FIG. 13 is now described with reference to the flowchartin FIG. 32.

In Step S441, the B component image calculation processing unit 82selects one unprocessed notice pixel.

In Step S442, the B component image calculation processing unit 82detects the B pixels existing within the predetermined range from thetarget pixel, wherein the predetermined range indicates a rangeexperientially or experimentally determined based on the conditions suchas the number of samples used for the principle component analysis andthe closeness of correlation depending on a distance from the targetpixel. Assuming that the target pixel position is represented as thecoordinates (H, K), for instance, the predetermined range may includethose such as a range of a size of 9×9 pixels represented by thecoordinates (X, Y), where H−4≦X≦H+4 and K+4≦Y≦K+4. Alternatively, arange having a different size or a range covering the predeterminednumber of pixels selected from the pixels existing close to each otherwith the target pixel as the center may also be used.

Assuming that the target pixel position is represented as thecoordinates (H, K), for instance, the pixels indicated by B exist atpositions represented by the coordinates (X, Y) where H−4≦X≦H+4,K+4≦Y≦K+4, X is odd-numbered, Y is even-numbered and X−Y−1 takes amultiple of 4.

In Step S443, the B component image calculation processing unit 82acquires the G pixel value estimated through the processing of the StepS51 in FIG. 3 with respect to all the detected B pixel (X, Y) positions.Specifically, the B component image calculation processing unit 82acquires more than one pair of B and G pixel values with respect to allthe detected B pixel (X, Y) positions.

In Step S444, the B component image calculation processing unit 82performs the principle component analysis after plotting the pairs of Band G pixel values on the two-dimensional space of B and G, as shown inFIG. 33.

In this processing, when it is assumed that the variance and the averageof the B pixel values within the predetermined range are respectivelyrepresented as varianceB and AveB, and that the variance and the averageof the G pixel values with respect to these B pixel positions arerespectively represented as varianceG and AveG, an equation of astraight line of the first principle component is shown as the followingequation (8).B=S×√{square root over (variance B)}÷√{square root over (varianceG)}×(G−AveG)+AveB   (8)where a coefficient S is assumed to be +1 when the correlation ispositive, and to be −1 when the correlation is negative.

In Step S445, the B component image calculation processing unit 82obtains the B value based on the straight line resulting from theprinciple component analysis, together with the G value with respect tothe target pixel. Specifically, the B component image calculationprocessing unit 82 calculates the B value by substituting the G pixelvalue with respect to the target pixel in the equation (8).

In Step S446, the B component image calculation processing unit 82judges whether or not the B value has been already obtained for all thepixels. When the result of judgment in the Step S446 is that the B valueis not obtained yet for all the pixels, the processing returns to theStep S441, causing reprocessing to be performed from the Step S441. Whenthe result of judgment in the Step S446 is that the B value has beenalready obtained for all the pixels, the processing returns to the StepS52 in FIG. 13, and is followed by the Step S53.

The processing like the may provide the B component pixel value for allthe pixels based on the pixel value of the B component contained in theimage data, together with the G pixel value estimated through theprocessing of the Step S51 in FIG. 13.

In this processing, it is also noted that the processing is put intopractice assuming that the B and G pixel values are correlated to eachother within the predetermined range such as those where H−4≦X≦H+4 andK+4≦Y≦K+4, for instance.

The respective processing has been described as related to oneprocessing of obtaining the pixel data of the spectral component (or theG (green) component, for instance) in the predetermined frequency rangehaving no chromatic aberration by subtracting the pixel (or the pixelindicated by M, for instance) data of the second pixel group having thesecond spectral component data from the pixel (or the pixel indicated byW, for instance) data of the first pixel group having the first spectralcomponent data (in other words, by calculating the linear sum of thefirst pixel group and the second pixel group based on the weights W1 andW2 where the weight W1=1 and the weight W2=(−1). Specifically, the hasbeen described on the case where the equation of G=W−M holds. On theother hand, if M′ whose sensitivity is as low as ½ of the sensitivity ofthe M pixel is used, for instance, the equation of G=W−M′ is not true.Thus, in this case, it suffices to perform the demosaic processing basedon the equation of G=W−2×M′, instead of the. Specifically, it is alsoallowable to perform the same processing as the by making an attempt togain up an output value of the pixel corresponding to M in thesolid-state imaging device 23 as much again, before supplying to the A/Dconverting unit 24. When the sensitivity of a signal corresponding toany one of the color components is inferior to that of other signals, itis also allowable to perform the same processing as the by making anattempt to gain up, at a predetermined scale factor, an output value ofthe corresponding pixel in the solid-state imaging device 23, beforesupplying to the A/D converting unit 24, likewise.

In other words, the present invention is applicable to cases where thefirst spectral component frequency band or the second spectral componentfrequency band is one that is not described above, or consists of morethan one non-continuously separated frequency bands, for instance, aslong as that an image of frequency range having no chromatic aberrationand the demosaic processing is performed using the image as reference,based on the signal component corresponding to the spectral component inthe predetermined frequency range resulting from subtracting the signal(or the value resulting from multiplying this signal by a predeterminedfactor) in the frequency band of the second spectral component from thesignal (or the value resulting from multiplying this signal by apredetermined different factor) in the frequency band of the firstspectral component. Further, an agreement of the spectral component inthe predetermined frequency range resulting from subtracting the secondspectral component frequency band from the first spectral componentfrequency band with the range of the frequency having no chromaticaberration, like the processing, preferably provides simplifiedprocessing. For instance, the present invention is also applicable, evenif attempting to assume the second spectral component frequency band tobe the infrared component where the first spectral component frequencyband is specified as the G (green) component and the infrared component,or to assume the second spectral component frequency band to be the R(red) component and the B (blue) component where the first spectralcomponent frequency band is specified as the visible light componentexcept the infrared component.

The application of the present invention as described above enables tocancel lens' chromatic aberration and to perform the demosaicprocessing, while holding the pixels that are capable of acquiring thewide wavelength-band spectral component.

It should be noted that setting of the color filter array may include anarray structure other than one shown in FIG. 7, and an array shown inFIG. 34 may also be acceptable. The color filter array shown in FIG. 34is equivalent to the array structure resulting from turning the arrayshown in FIG. 7 by 45 degrees.

Specifically, with respect to the color filter shown in FIG. 34,referring to the pixels indicated by G assuming that the X-axisdirection is set as the 0-degree direction, the pixels indicated by Gare disposed in every other pixel in all of the 0-degree direction, the45-degree direction, the 90-degree direction and the 135-degreedirection, wherein both the pixels adjacent in the 0-degree direction tothe pixel indicated by G are of pixels indicted by B or R, both thepixels adjacent in the 90-degree direction to the pixel indicated by Gare of pixels indicated by B or R, both the pixels adjacent in the45-degree direction to the pixel indicated by G are of pixels indictedby M, and both the pixels adjacent in the 135-degree direction to thepixel indicated by G are of pixels indicated by W.

It should be noted that with respect to the color filter array shown inFIG. 34, the reversed arrangement of the W and M pixels may also allowto perform the demosaic processing in the same manner. In addition, thereversed arrangement of the B and R pixels may also allow to perform thedemosaic processing in the same manner.

Specifically, the use of the color filter of a type that referring tothe pixels indicated by G assuming that the X-axis direction is set asthe 0-degree direction, the pixels indicated by G are disposed in everyother pixel in all of the 0-degree, 45-degree, 90-degree and 135-degreedirections, and other color component pixels, that is, the R, B, W and Mpixels, are arranged adjacent in the respective directions so that the Rand B pixels and the Wand M pixels are respectively located orthogonalto the G pixels are adaptable to perform the demosaic processing in thesame manner.

It should be noted that, in the present embodiment, the demosaicprocessing is performed by obtaining the G component image based on theinterpolation processing in the respective directions after calculatingthe pattern probability in the directions at four angles, 0, 45, 90 and135 degrees. Alternatively, the present invention may also be applicableto processing which uses directions other than ones at four angles, 0,45, 90 and 135 degrees. It is obvious that the present invention is alsoapplicable to the processing with respect to directions at two angles,such as a pair of 0-degree direction and 90-degree direction, and a pairof 45-degree direction and 135-degree direction.

Typically, capturing of a landscape image and the like frequentlyresults in an image containing the patterns in the 0-degree directionand the 90-degree direction with respect to an obtained image, in whichcase, however, there are not so much patterns in the 45-degree directionor the 135-degree direction, as compared with the patterns in the0-degree and 90-degree directions. Thus, in order to reduce a circuitsize or save an operation time, it is also allowable to perform thedemosaic processing by, after calculating the pattern probability in thedirections at two angles, that is, the 0-degree direction and the90-degree direction, obtaining the G component image based on theinterpolation processing in the respective directions.

The processing among the series of processing, particularly, theprocessing attained with the camera signal processing unit 25 is notlimited to those by hardware; and it is also allowable to apply softwareto perform the processing. The software is installed, through therecording medium and the like, into a computer such as a computer whosededicated hardware is integral with a program contained in the softwareand a general-purpose personal computer, for instance, adaptable toimplement various functions through installation of various programs. Inthis case, the image-capture apparatus 11 previously described withreference to FIG. 4 includes a personal computer 101 as shown in FIG.35, for instance.

Referring to FIG. 35, a CPU (Central Processing Unit) 111 performsvarious processing according to a program contained in a ROM (Read OnlyMemory) 112 or a program loaded from a storage unit 118 into a RAM(Random Access Memory) 113. Data such as those required for the CPU 111to perform the various processing is also stored in the RAM 113according to circumstances.

The CPU 111, the ROM 112, and RAM 113 are interconnected through a bus114. An input/output interface 115 is also connected to the bus 114.

An input unit 116 including units such as a keyboard and a mouse, anoutput unit 117 including units such as a display and a speaker, thestorage unit 118 including units such as a hard disk, a communicationsunit 119 including units such as a modem and a terminal adapter, and animage-capture processing unit 120 are all connected to the input/outputinterface 115. The communications unit 119 performs communicationprocessing over a network including Internet.

The image-capture processing unit 120 is in the form of a unit havingthe optical lens 21, the color filter 22, the solid-state imaging device23 and the A/D converting unit 24 previously described with reference toFIG. 4 or a unit adaptable to implement the same functions as the units,and performs the same processing as the under control of the CPU 111having the functions of the camera signal processing unit 25 previouslydescribed with reference to FIG. 5 and the image compressing unit 27shown in FIG. 4.

A drive 121 is also connected to the input/output interface 115 at need,and is fitted with the recording medium such as a magnetic disk 111, anoptical disc 112, a magneto-optical disc 133 and a semiconductor memory134 according to circumstances, permitting a computer program read outfrom the recording medium to be installed into the storage unit 118 atneed.

When an attempt is made to adapt the software to perform the series ofprocessing, the software is installed through the network and/or therecording medium into a computer such as a computer whose dedicatedhardware is integral with a program contained in the software concerned,and a general-purpose personal computer adaptable to implement variousfunctions through installation of various programs.

The recording medium includes not only program-contained package mediasuch as the magnetic disk 111 (including floppy disks), the optical disc112 (including CD-ROMs (Compact Disk-Read Only Memories) and DVDs(Digital Versatile Disks), the magneto-optical disc 133 (including MDs(Mini-Disks: Trade Mark) and the semiconductor memory 134 or thosedistributed separately from the apparatus body to supply the program tothe users, but also the hard disks contained in the ROM 112 or thestorage unit 118 containing the program or those supplied in the form ofapparatus body-integrated media to the users.

In the present specification, the steps indicating the program recordedin the recording medium includes not only the processing that isperformed in time series and in the order listed, but also theprocessing that is not always in time series but may be in parallel orindividual manner.

The present application contains subject matters related to JapanesePatent Application No. 2006-213539 filed in Japanese Patent Office onAug. 4, 2006, the entire content of which being incorporated herein byreference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations, and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of appended claims and equivalents thereof.

1. A color filter allowing a light signal to pass through by each pixeland be incident on an imaging device, the light signal being inputtedthrough a lens and including one of plural different spectralcomponents, wherein: the plural different spectral components include: afirst spectral component which has a widest frequency bandwidth amongthe plural different spectral components, a second spectral componentcorresponding to a predetermined frequency band close to a frequencythat causes no chromatic aberration of the lens, and a third spectralcomponent expressed in terms of a linear sum of a value resulting frommultiplying the first spectral component by a first weighting factor anda value resulting from multiplying the second spectral component by asecond weighting factor, and wherein the plural different spectralcomponents are of five types of spectral components including the firstspectral component, the second spectral component and the third spectralcomponent, and pixels corresponding to the first spectral component andthe third spectral component and pixels corresponding to a fourthspectral component and a fifth spectral component are respectivelyarranged adjacent to a pixel corresponding to the second spectralcomponent in one of a 0-degree direction, a 45-degree direction, a90-degree direction and a 135-degree direction assuming that one arraydirection of a plane, on which the pixels are arrayed, is set as the0-degree direction, the pixels corresponding to the first spectralcomponent and the third spectral component are arrayed in a directionorthogonal to the pixel corresponding to the second spectral component,and the pixels corresponding to the fourth spectral component and thefifth spectral component are arrayed in the 45 degree or the 135 degreedirection from the pixel corresponding to the second spectral component.2. The color filter according to claim 1, wherein the first spectralcomponent includes at least an infrared component.
 3. The color filteraccording to claim 2, wherein the first spectral component includes theinfrared component and all frequency bands of visible light.
 4. Thecolor filter according to claim 1, wherein the first spectral componentincludes all frequency bands of visible light.
 5. The color filteraccording to claim 1, wherein the second spectral component is afrequency component having a predetermined range that corresponds to agreen component.
 6. The color filter according to claim 1, wherein thethird spectral component is a spectral component resulted by excludingthe second spectral component from the first spectral component.
 7. Thecolor filter according to claim 1, wherein pixels corresponding to thesecond spectral component are disposed in every other pixel in all of a0-degree direction, a 45-degree direction, a 90-degree direction and a135-degree direction assuming that one array direction of a plane, onwhich the pixels are arrayed, is set as the 0-degree direction.